OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Setup & Installation => Topic started by: roberto on January 03, 2023, 10:32:49 am

Title: Virtual memory, a curious phenomenon
Post by: roberto on January 03, 2023, 10:32:49 am
There is a strange result in virtual memory, depending on where the Swapper position is defined in the config.
In my case if I have the config.sys:
SWAPPATH=C:\OS2\SYSTEM 0 4096
after start the system:
[C:\]mem /v

Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:     514 MB   <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB

 But if I create a new unit in 512MB memory,with Ramdisk, unit L: \, HPFS type, and define in the config.sys as:
 SWAPPATH=L:\ 0 4096
 [C:\]mem /v

Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:   3 321 MB  <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB

It seems that the 512MB is more important, than anything else, so that it occurs, that change.
On the other hand I do not see any change in the rest of the operation of the system, but it took a short time of evidence.

Saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: ivan on January 03, 2023, 01:30:06 pm
I can never understand your setup.  I have ArcaOS v5.0.7 as a standard setup with c a couple of exceptions, the boot drive C: is hpfs and the swap file is on D:  I get

[C:\]mem /v

Total physical memory:      7,906 MB
Accessible to system:       3,311 MB
Additional (PAE) memory:    4,595 MB

Resident memory:              193 MB
Available virtual memory:   2,590 MB

Available process memory:
  Private low memory:         213 MB
  Private high memory:      2,240 MB
  Shared low memory:          149 MB
  Shared high memory:       2,160 MB

[C:\]

  SWAPPATH=D:\ 2048 2048

My test setup on my Ryzen third generation processor is very much the same.
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on January 03, 2023, 03:40:04 pm
Hello Ivan
You already have half of the test done.

Now you run this program:
C:\sys\bin\HIDISK.EXE
Define 512MB, HPFS, and Format US HPFS
Name unit X:
Change your config.sys
rem SWAPPATH=D:\ 2048 2048
SWAPPATH=X:\ 2048 2048
Restart the system
and
run mem /v
saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Dariusz Piatkowski on January 04, 2023, 02:10:08 am
...
Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:   3 321 MB  <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB
...

The 'Available virtual memory' you're seeing is what I have here on a 8G = 4G + 4G RAMDISK.

What I think is uber strange is that 'Shared high memory' reading!!! wwwwhhhhaaattt???? isn't that an OS/2 dream for everyone...seriously, the LOW/SEGMENTED shared memory area is our weak point!!!
Title: Re: Virtual memory, a curious phenomenon
Post by: Dave Yeo on January 04, 2023, 03:16:49 am
...
Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:   3 321 MB  <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB
...

The 'Available virtual memory' you're seeing is what I have here on a 8G = 4G + 4G RAMDISK.

What I think is uber strange is that 'Shared high memory' reading!!! wwwwhhhhaaattt???? isn't that an OS/2 dream for everyone...seriously, the LOW/SEGMENTED shared memory area is our weak point!!!

Hi Dariusz, there's something weird about your mem -v output if you have 8GB installed, your "Total physical memory" should show close to 8GB with "Additional (PAE) memory" showing about 4GB.
Here with 12 GB of ram installed, and a 8GB or so ram disk (JFS)
Code: [Select]
Total physical memory:     12,169 MB
Accessible to system:       3,241 MB
Additional (PAE) memory:    8,928 MB

Resident memory:              153 MB
Available virtual memory:   1,855 MB

Available process memory:
  Private low memory:         294 MB
  Private high memory:      2,240 MB
  Shared low memory:          177 MB
  Shared high memory:         980 MB

As you can see, my "Available virtual memory" does not reflect upper memory. I don't really understand virtual memory excepting it includes ram plus swap file space and it is changable depending on various things like when I built Qt5 and the system was swapping (yes I hit the swap file with 3.2GB of ram visible), it likely dropped to close to zero.
As for Shared high memory, that is where DLL's go when marked to load high, I have Dooble and SeaMonkey open right now so as you see, "Shared high memory" is down to 980 MB. Closing Dooble gives,
Code: [Select]
Total physical memory:     12,169 MB
Accessible to system:       3,241 MB
Additional (PAE) memory:    8,928 MB

Resident memory:              152 MB
Available virtual memory:   1,973 MB

Available process memory:
  Private low memory:         294 MB
  Private high memory:      2,240 MB
  Shared low memory:          193 MB
  Shared high memory:       1,175 MB

More virtual memory now as well as shared low/high memory.
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on January 04, 2023, 12:54:17 pm

Dariusz- The important thing for me is not to compare a computer with another. The important thing is to compare the results on the same computer.

Dave- After several days trying, the improvement is clear for me. The sensation is that the Swapper was not working well, and now if he is doing it correctly. What I cannot understand, is because it is so important that the amount of 512MB, of space in the Ramdisk, makes the swapper improve. I tried it with 64mb and it didn't work. I don't know what happens with other Ramdisk unit values.
I compare the two results after starting the same computer, and without having started any other application. Except for the applications that are started in the startup, but in both cases they are the same.
Saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Roderick Klein on January 04, 2023, 09:21:23 pm
There is a strange result in virtual memory, depending on where the Swapper position is defined in the config.
In my case if I have the config.sys:
SWAPPATH=C:\OS2\SYSTEM 0 4096
after start the system:
[C:\]mem /v

Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:     514 MB   <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB

 But if I create a new unit in 512MB memory,with Ramdisk, unit L: \, HPFS type, and define in the config.sys as:
 SWAPPATH=L:\ 0 4096
 [C:\]mem /v

Total physical memory:      3 895 MB
Accessible to system:       2 999 MB
Additional (PAE) memory:      896 MB

Resident memory:              129 MB
Available virtual memory:   3 321 MB  <------

Available process memory:
  Private low memory:         373 MB
  Private high memory:      2 240 MB
  Shared low memory:          310 MB
  Shared high memory:       2 211 MB

It seems that the 512MB is more important, than anything else, so that it occurs, that change.
On the other hand I do not see any change in the rest of the operation of the system, but it took a short time of evidence.

Saludos

Sorry but the location of the swappath in the config.sys has no influence when the swapper gets set up.
This is most likely one of the kernel statements that gets processed very early by the kerne.
DEVICE= statement for example depend on the order in the config which driver is loaded.
I mean the location of the libpath line in the config.sys also does not alter that behaviour.

Roderick
Title: Re: Virtual memory, a curious phenomenon
Post by: Rich Walsh on January 04, 2023, 09:40:07 pm
I just did a lot of testing on this. I found that the value shown for "Available virtual memory" depends on the size of your swap partition and nothing else. If your swap partition is less than at least 4gb, then the value shown is likely to be useless.

Here are the numbers for my normal setup using  "SWAPPATH=N:\ 2048 2048", where N: is the 4.9gb boot drive.

Code: [Select]
Total physical memory:     16,171 MB
Accessible to system:       2,355 MB
Additional (PAE) memory:   13,816 MB

Resident memory:              130 MB
Available virtual memory:   1,215 MB

Available process memory:
  Private low memory:         379 MB
  Private high memory:      1,792 MB
  Shared low memory:          347 MB
  Shared high memory:       1,763 MB

Moving the swappath to a 1gb ramdisk _or_ harddrive gives me:
Code: [Select]
Available virtual memory:   3,157 MB

Increase the partition to 2gb and I get:
Code: [Select]
Available virtual memory:      67 MB

Move the swappath to a 50gb harddrive:
Code: [Select]
Available virtual memory:      2,179 MB

Turn off swapping (MEMMAN=NOSWAP,PROTECT):
Code: [Select]
Available virtual memory:      2,101 MB

The values that you see here are returned by a call to DosQuerySysInfo(QSV_TOTAVAILMEM). I don't think anyone has ever questioned its output before, but it looks like there is good reason to believe it's buggy (or at least _very_ counter-intuitive).
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on January 05, 2023, 06:31:23 pm
Hello
The virtual memory, from what I see is rotary, goes from 0 to 4GB and 4GB to 0, constantly.
Therefore, it begins with an amount or another can be the same in theory. But while towards the tests
I have seen how with certain values, the keyboard was blocked for several minutes.
The latter as it has been after restarting several times, I don't know if it can be a different problem.

My results are very similar to yours, 1GB gives me more free, 2GB gives me less free.

For a while I will use the 512MB, and see how it works for me.

Saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Dariusz Piatkowski on January 06, 2023, 03:29:00 am
Dave, everyone...
Hi Dariusz, there's something weird about your mem -v output if you have 8GB installed, your "Total physical memory" should show close to 8GB with "Additional (PAE) memory" showing about 4GB.
...

Not sure what happened there, other than a few neurons clearly missfiring on my part...grrh...sorry, my post is of course blatantly incorrect. Thank you for pointing that out.
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on January 26, 2023, 03:32:35 pm
-Roderick
       It is not the same as the swapper put it in different places c:\ c:\os2 \  d: \ etc.
        To the size of the disc where you put it, is different.
        I think that has already been clear in this thread.

-Dariusz
       Do not apologize, the important thing is the interest and curiosity for things,
       keep it up.

Now I am going to tell you a real case, Arcaos 507 new, recently installed in a Lenovo laptop
Ideapad z710 with virtualaddreslimit = 1536 all default

As soon as I start the laptop ,this is memory, I couldn't even start the Seamonkey ¿¿???
[C:\]mem /v

Total physical memory:      8 124 MB
Accessible to system:       2 502 MB
Additional (PAE) memory:    5 622 MB

Resident memory:              134 MB
Available virtual memory:   2 322 MB

Available process memory:
  Private low memory:         257 MB
  Private high memory:        896 MB
  Shared low memory:          166 MB
  Shared high memory:         875 MB

[C:\]

AFTER  ramdisk 512mb and and modify the config.sys changing the swapper to that newly created partition

[C:\]mem /v

Total physical memory:      8 124 MB
Accessible to system:       2 502 MB
Additional (PAE) memory:    5 622 MB

Resident memory:              130 MB
Available virtual memory:   2 331 MB

Available process memory:
  Private low memory:         380 MB  -------257--------
  Private high memory:        896 MB
  Shared low memory:          320 MB  ------166---------
  Shared high memory:         875 MB

Here you can see Swapper's magic in 512MB, memory increases.
[C:\]

AFTER virtualaddress 3072  and restart

[C:\]mem /v

Total physical memory:      8 124 MB
Accessible to system:       2 502 MB
Additional (PAE) memory:    5 622 MB

Resident memory:              130 MB
Available virtual memory:   2 332 MB ----2331-------
Available process memory:
  Private low memory:         380 MB
  Private high memory:      2 240 MB ----896-------
  Shared low memory:          321 MB ----320------
  Shared high memory:       2 219 MB ----875------


If this would be on a computer, it could be just something curious, but at work I had long
With this modification, and it is clearly not the same, in all and seen something very similar.
With an important improvement of functions and performance.
As it is very simple to do it, anyone can try it and will be before and after

Saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on January 27, 2023, 02:58:28 pm
Here is what I have using my settings  (I changed virtualaddresslimit until I could have max free available memory, other value provides less usable memory)

VIRTUALADDRESSLIMIT=2944
SWAPPATH=M:\ 131072 524288
M is my virtual DRIVE using all PAE memory                 

Total physical memory:      16262 MB
Accessible to system:        3470 MB
Additional (PAE) memory:    12792 MB

Resident memory:              215 MB
Available virtual memory:    2412 MB

Available process memory:
  Private low memory:         282 MB
  Private high memory:       2128 MB
  Shared low memory:          193 MB
  Shared high memory:        1329 MB


Note: this doesn't prevent me having mouse freeze from time to time but it more seems due WPS with not refreshed screen during the freeze time (suspect png despite I have all png updates installed supposed to correct this kiond of problems (reduces may I say) ! - opening a very bigfolder with png show a dramatic free shared storage decrease and sometimes, when not release at folder close, a WPS restart frees some memory but not all)
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on January 28, 2023, 01:45:42 pm
-Remy
     
If you have no more available memory, it's because you don't want to.
Or that I have not explained myself well,or....
You do not need to reinstall the entire system, just from a good position.
the procedure is the next:(I understand that I do not indicate the installation of your system must be by default.)
Try to do this and tell me how it run:
SWAPPATH=C:\ 2048 2048 or something similar
Virtualaddrreslimit = 1536
Restart the system
Create a Disk RAM with 512MB HPFS, and PUT the swappath in that newly created unit, swappath=M:\ 2048 2048
Restart Again
Now virtualaddresslimit = 3072
Restart and test.

Saludos
PD: The grass that smokes must be very good, my whiskey changed for your grass,( just kidding )

Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on January 30, 2023, 12:06:07 am
-Remy
     
If you have no more available memory, it's because you don't want to.
Or that I have not explained myself well,or....
You do not need to reinstall the entire system, just from a good position.
the procedure is the next:(I understand that I do not indicate the installation of your system must be by default.)
Try to do this and tell me how it run:
SWAPPATH=C:\ 2048 2048 or something similar
Virtualaddrreslimit = 1536
Restart the system
Create a Disk RAM with 512MB HPFS, and PUT the swappath in that newly created unit, swappath=M:\ 2048 2048
Restart Again
Now virtualaddresslimit = 3072
Restart and test.

Saludos
PD: The grass that smokes must be very good, my whiskey changed for your grass,( just kidding )

Hi Roberto,

I already tested several settings with all giving me different results like you found too.
The setting I posted is the one providing me the most usable storage and the most impacting parameter was
virtualaddresslimit (the value you suggest decreased usable storage and I found that setting it to 2944 was better than 3072).

Compare my results with virtualaddresslimit=2944 with yours virtualaddresslimit=3072 !
Any usual vitualaddresslimit 1536, 2048, 2560, 3072 .... produces worst results.   
                             
Total physical memory:      16262 MB
Accessible to system:        3470 MB         <<<<<   with 3072, you have 2502 ! (no change compared to 1536 !)

Resident memory:              215 MB           <<<<<   with 3072, you have 130 !  (no change compared to 1536 !) 
Available virtual memory:    2412 MB        <<<<<   with 3072, you have 2332 !  (no change compared to 1536 !)


I agree with you that results are strange like you described which made me search which value allow me to get the most usable storage. Give a try with virtualaddresslimit=2944 instead of 3072 and check mem output.
See mem results I have.

Is the mem output accurate ?
Title: Re: Virtual memory, a curious phenomenon
Post by: Dave Yeo on January 30, 2023, 05:05:16 am
Is the mem output accurate ?

Not an expert but I assume that it accurately reports what the kernel reports. Which leads to the question of whether the kernel's report is accurate. Judging by the wildly different results, it doesn't seem to be, likely as when it was designed having 4GB's of ram installed was a fantasy and before Warp V4.5 (Warp V4+fp 13) the non-server OS/2 could only access 1GB of address space, half of which was given to the kernel. That is low memory.
We're dealing with different types of memory.
Accessible to the system is how much actual ram is accessible, 4GB minus PCI space and such including video memory. I couldn't even install ArcaOS to this box until I reduced the video ram (onboard graphics). Now it is about 3.4 GB's, I'm not in OS/2 right now so going by memory. My usual box, it is 3.2 GB.
Virtual memory is how much the CPU can access, you can have multiple processes using the same address range like with multiple DOS sessions. IIRC, the virtual limit, in theory, is 2TB. As it allocated in 4KB chunks, there's selectors and such to keep track of it and they'll run out. The Kernel also has to keep track and uses 32 bit variables, sometimes signed, so 2-4GB limit before perhaps wrapping or overflowing and crashing. Virtual memory can live in the swap file as well as ram. Our swap file is limited to 2GB, I've had the system die due to a swap file full condition when it grew past 2GB.
In theory, you could start a bunch of DOS sessions, each with access and committing 1GB of memory and once ram is full, it'll swap. In practice, those 32bit variables limit things.
I think VIRTUALADDRESSLIMIT is how much address space a (32 bit) process can use, so the higher the better, at least until it interferes with kernel (device drivers etc) and PCI space.  My feeling is that IBM didn't quite finish things so we have to manually set it.
Then there is low and high memory, 16 bit programs are limited to 1GB of virtual memory, mapped to the lower 1GB, half of which is claimed by the kernel so 512MB of low memory, shared and private with the shared having a bunch of DLL's loaded before you even launch a program. So basically shared is DLL space.
I think each process gets its own private address space as well but I'm not sure.
Anyways, that's my understanding, much of it might be wrong, maybe someone can correct me but the bottom line is the most important part of the mem output is shared memory, especially low shared memory as that is what we usually run out of first.
It is possible to run out of address space as well. Linking xul.dll (Firefox) for a while would fail without virtualaddresslimit set to 3072, and with 1.5 Gb of real ram, I had to be careful not to run anything else as the swap file would overflow, now some of the Qt DLLs and source code seem to have the same problem and with 3.2GB's of ram accessible to the system, building Qt5, I saw the swap file grow by a few hundred MB's, not sure how big as I wasn't paying attention and it likely shrank before I noticed. Also have to limit the make jobs that are running, 2 instead of 5 on this 4 core system.
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on January 30, 2023, 05:48:14 pm
Is the mem output accurate ?

I think VIRTUALADDRESSLIMIT is how much address space a (32 bit) process can use, so the higher the better, at least until it interferes with kernel (device drivers etc) and PCI space.  My feeling is that IBM didn't quite finish things so we have to manually set it.
Then there is low and high memory, 16 bit programs are limited to 1GB of virtual memory, mapped to the lower 1GB, half of which is claimed by the kernel so 512MB of low memory, shared and private with the shared having a bunch of DLL's loaded before you even launch a program. So basically shared is DLL space.
I think each process gets its own private address space as well but I'm not sure.


It looks to be like this, OS/2 was build using ideas out of IBM MVS.
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on January 30, 2023, 06:31:32 pm
Hi Remy

In my work computer with 2944 virtualaddressl ... A improved, it is modern.
In my personal laptop with 2944, it is worse, it is old but very well built. But comment, in this equipment, I use the screen pre-actualice the video in intermediate memory,
And it consumes me 8Mb. and with 3072 the screen was corrupted, and left the 3072 without pre -actualizing the screen,
or I get down to 3064 and preactive the panatalla. (This better option)

I have two questions, how did you calculate the value 2944?
In the swappath that you had previous 52428/131072 = 4 exact, you had same system to calculate it,
Was it for proof and error?

I understand that programers have to adjust many things, including the low memory, which is very annoying,
When you tell you that you lack memory, and you have a lot unused, of another type.

I guess we will have to start playing with the SWAPPATH.
Saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Dave Yeo on January 30, 2023, 07:22:32 pm
Hi Roberto,
If you use hex for virtualaddresslimit, you see the proper values end in 00, which is a page boundary.
3072=C00h
2816=B00h
2560=A00h
2304=900h
2048=800h
1792=700h
1536=600h

it is likely important to have a page boundary such as above.
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on January 30, 2023, 08:08:29 pm
I really don't know what Page Boundary is, but I know what works for me and what doesn't. It seems to me that with 3064 it is going well, but it is soon to say anything, I need more use to verify it.
What I am starting to think is that SwaPPath is a frequency and not an amount, which generates a Swapper file.
I would like to try this:
Code: [Select]
swappath=L:\ 2000 506000

saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on January 30, 2023, 09:10:27 pm
I have already understood Page Boundary, I learn quickly. I understand that Remy's 2944 are out of rank, and can be dangerous.
Thanks dave
Saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Dave Yeo on January 30, 2023, 10:00:32 pm
I have already understood Page Boundary, I learn quickly. I understand that Remy's 2944 are out of rank, and can be dangerous.
Thanks dave
Saludos

I guess it depends on how things are implemented, 2944=B80h, still a fairly round number, remembering that 80h is half of 100h.
Personally, I'd stick with the the recommended values. Could use Theseus to look closer.
Really this is just playing with numbers, the important thing is how much memory can actually be allocated and committed, and when it matters.
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on January 31, 2023, 02:39:18 am
I have already understood Page Boundary, I learn quickly. I understand that Remy's 2944 are out of rank, and can be dangerous.
Thanks dave
Saludos

I guess it depends on how things are implemented, 2944=B80h, still a fairly round number, remembering that 80h is half of 100h.
Personally, I'd stick with the the recommended values. Could use Theseus to look closer.
Really this is just playing with numbers, the important thing is how much memory can actually be allocated and committed, and when it matters.

Well, just refresh me, a page is 4K, right ?
2944 is a multiple of 4 ( 4 *  736 ) or ( 64 * 46 )
When I use any of suggest values, my usable storage is always dropping to a low value, I did approach using 4k pages and this value was the one providing me with the maximum available storage.
Hi Roberto,
If you use hex for virtualaddresslimit, you see the proper values end in 00, which is a page boundary.
3072=C00h
2816=B00h
2560=A00h
2304=900h
2048=800h
1792=700h
1536=600h

it is likely important to have a page boundary such as above.


Why do you use a MiB converted to hex (MiB value) to determine page boudary ?
I though a page is 4KiB, is it ? 
1536 is MiB or 1572864 KiB  ( 180000H ) - number of 4k pages in hex value
2944 -> 3014656 KiB ( 2E0000H ) - number of 4k pages in hex value

OS/2 is able to use up to near 3.5GiB, could you get a value near this one using your settings ?
I tried several setting in respect of 4k pages and I could see that using 3072, the system rounded the storage to a lower value. I think (should be confirmed) that OS/2 is calculating the value to satisfy boundaries pages and despite the given value, it changes it to satisfy its architecture and the mem output is the value OS/2 determined to be correct according the given limit.

It is correct ?
I'm interested to get more details about how it works here.   
 
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on January 31, 2023, 02:55:19 am
Hi Remy

In my work computer with 2944 virtualaddressl ... A improved, it is modern.
In my personal laptop with 2944, it is worse, it is old but very well built. But comment, in this equipment, I use the screen pre-actualice the video in intermediate memory,
And it consumes me 8Mb. and with 3072 the screen was corrupted, and left the 3072 without pre -actualizing the screen,
or I get down to 3064 and preactive the panatalla. (This better option)

I have two questions, how did you calculate the value 2944?
In the swappath that you had previous 52428/131072 = 4 exact, you had same system to calculate it,
Was it for proof and error?

I understand that programers have to adjust many things, including the low memory, which is very annoying,
When you tell you that you lack memory, and you have a lot unused, of another type.

I guess we will have to start playing with the SWAPPATH.
Saludos

SWAPPATH=M:\ 131072 524288

About my swappath values, I set numbers very high because I have a big PAE disk. a high allocated value reduces allocation processes into os/2 when it uses the swap file. The other value is the mini I decided to not go below which gives me time to have an action before "damage" :D
e.g. 512 * 1024 (1K binary) =>   524288 and  124 * 1024 = 131072   (I think that a good approach about the initial swappath value would be something near the maximum swappath size used - of course, this has more impact when using the swappath on a HD but reducing processes like increase swappath seems to me to be wellcome despite it is into memory)     
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on January 31, 2023, 03:06:12 am
The lack of memory I have occurs below 512M and often when handling big and many png image files (WPS uses a lot too !) despite having the latest fixes for png supposed correcting this, the problem is reduced but not resolved. WPS is an other one consuming low storage and not always releasing it (WPS restart will get back parts of the storage)   
Other programs have they dlls loaded into the high storage and I have enough storage there.
Title: Re: Virtual memory, a curious phenomenon
Post by: Dave Yeo on January 31, 2023, 07:06:55 am
I have already understood Page Boundary, I learn quickly. I understand that Remy's 2944 are out of rank, and can be dangerous.
Thanks dave
Saludos

I guess it depends on how things are implemented, 2944=B80h, still a fairly round number, remembering that 80h is half of 100h.
Personally, I'd stick with the the recommended values. Could use Theseus to look closer.
Really this is just playing with numbers, the important thing is how much memory can actually be allocated and committed, and when it matters.

Well, just refresh me, a page is 4K, right ?
2944 is a multiple of 4 ( 4 *  736 ) or ( 64 * 46 )
When I use any of suggest values, my usable storage is always dropping to a low value, I did approach using 4k pages and this value was the one providing me with the maximum available storage.
Hi Roberto,
If you use hex for virtualaddresslimit, you see the proper values end in 00, which is a page boundary.
3072=C00h
2816=B00h
2560=A00h
2304=900h
2048=800h
1792=700h
1536=600h

it is likely important to have a page boundary such as above.


Why do you use a MiB converted to hex (MiB value) to determine page boudary ?
I though a page is 4KiB, is it ? 
1536 is MiB or 1572864 KiB  ( 180000H ) - number of 4k pages in hex value
2944 -> 3014656 KiB ( 2E0000H ) - number of 4k pages in hex value

My use of page was probably not the best, apologies. For the i386, a page is 4KiB, an i686+ can also have 1MiB pages, not that it happens with OS/2.
OS/2 2.x allocates in 4KiB chunks, Warp allocates 64KiB chunks, but might only commit 4KiB, so probably better to use 64KiB's in your calculations.
I used the page boundary converted to hex as computers work in binary which more accurately are represented in hex. For certain things such as fixup tables, round hex numbers are better. I (and most of us) don't know how things are programmed in the kernel when it comes to high memory, so I was guessing that it only works with some numbers such as I listed.
The documentation is lacking, looking now, I can only find mention of VIRTUALADDRESSLIMT in the programming addendum under DosQuerySysInfo(), which is the function that mem uses to get most of its info, it says,
Code: [Select]
30 QSV_VIRTUALADDRESSLIMIT

Size of the user's address space in megabytes (that is, the value of the rounded VIRTUALADDRESSLIMIT)
Not much to go on, besides what others say. I did test your setting of VIRTUALADDRESSLIMIT=2944,
Seems to work, From Theseus, General System Information,"QSV_VIRTUALADDRESSLIMIT  = b8000000"

Quote
OS/2 is able to use up to near 3.5GiB, could you get a value near this one using your settings ?
I tried several setting in respect of 4k pages and I could see that using 3072, the system rounded the storage to a lower value. I think (should be confirmed) that OS/2 is calculating the value to satisfy boundaries pages and despite the given value, it changes it to satisfy its architecture and the mem output is the value OS/2 determined to be correct according the given limit.

It is correct ?
I'm interested to get more details about how it works here.   

As I said, I'm not really sure if my understanding is correct. The documentation is clear that the values returned by DosQuerySysInfo() is only advisory,
Code: [Select]
19 QSV_TOTAVAILMEM

Maximum number of bytes of memory that can be allocated by all processes in the system. This number is advisory and is not guaranteed, since system conditions change constantly.

With similar for the others.
I'm really not sure how much some of these numbers such as available virtual memory matter as well as how accurate they are. Changing the size/location of the swap file shouldn't matter, yet it changes what the system reports, weird.
I guess what matters is getting out of memory errors and when it happens.
Building Qt5, I got failures of cpp.exe not being able to allocate enough memory, reboot fixed, I also got  popups about not enough resources to start make (lots of make processes run during the build), also fixed by a reboot, and I also saw my swap file grow, seems I used more memory then the 3.2 GiB accessible to my system.
Unluckily, a lot of the times, the system just locks up when memory gets low. Probably depends on the type of memory as things get funky when shared gets low. Builds do fail in similar ways, locks up and eventually have to reboot to get rid of the zombie process, reducing the number of jobs that make runs usually fixes that issue, as well as having VIRTUALADDRESSLIMIT set to 3072, which at one point wlink would complain and fail due to not enough spill memory (address range) without that setting.
Not an expert and I'm sure I've got some stuff wrong.
Title: Re: Virtual memory, a curious phenomenon
Post by: Dave Yeo on January 31, 2023, 07:21:14 am
The lack of memory I have occurs below 512M and often when handling big and many png image files (WPS uses a lot too !) despite having the latest fixes for png supposed correcting this, the problem is reduced but not resolved. WPS is an other one consuming low storage and not always releasing it (WPS restart will get back parts of the storage)   
Other programs have they dlls loaded into the high storage and I have enough storage there.

The problem is that 16 bit functions have to be in low memory, and there's a few important ones. Here's a partial list, too lazy to fight with github right now so this is outdated but gives the idea, you can look at the installed os2safe.h. [urlhttp://trac.netlabs.org/libc/browser/branches/libc-0.6/src/emx/include/os2safe.h[/url]
I don't know why png uses so much but the WPS and PMSHELL do use these functions and have to be in low memory. For libc programs, there's forwarders to load high and call low.
Another big issue is memory fragmentation. After a while of committing and releasing memory, it can get fragmented and even though you have 200 MB of free low shared memory, the biggest chunk might be quite a bit smaller so trying to allocate lets say 50 MB when the biggest hole is 40 MB is going to fail.
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on January 31, 2023, 09:05:08 am
Hi Dave!

I think it could be helpfull to read how it works into MVS (or just before MVS) because IBM used it's logics (how it should work) to build OS/2. In MVS, pages = 4K (of course, 4k and 64k are the most used values). Each init has an address space witha size which can be above the installed storage and up to the maximum addressable limit ... (very simplified view)  ::)
About pagination, I think there is a expansion streshold and a contraction streshold and It would be interesting to find if these streshold can be tunned or are these streshold hardcoded by architecture !

 ::) :-\
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on January 31, 2023, 09:12:59 am
Hi Remy

In my work computer with 2944 virtualaddressl ... A improved, it is modern.
In my personal laptop with 2944, it is worse, it is old but very well built. But comment, in this equipment, I use the screen pre-actualice the video in intermediate memory,
And it consumes me 8Mb. and with 3072 the screen was corrupted, and left the 3072 without pre -actualizing the screen,
or I get down to 3064 and preactive the panatalla. (This better option)

I have two questions, how did you calculate the value 2944?
In the swappath that you had previous 52428/131072 = 4 exact, you had same system to calculate it,
Was it for proof and error?

I understand that programers have to adjust many things, including the low memory, which is very annoying,
When you tell you that you lack memory, and you have a lot unused, of another type.

I guess we will have to start playing with the SWAPPATH.
Saludos

Hi roberto,

The value I use can be divided by 4 and 64 while yours only by 4 ! (I would not use)
May be you could try 3072, 3008, 2944, 2880 ...  (values I tried in the past) 
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on January 31, 2023, 06:10:28 pm
 REMY
Thanks for the virtualaddresslimit, proposed as possible.
I have only tried 3008, for my screen problem. And it doesn't work, the screen is corrupted.
In principle, the rest, and stay with those of a lifetime, proposed by Dave.

The Swappph, SwaPPath = m: \ 131072 524288, it seems that it works and is correct, but it is slow.
Generates this swapper.dat:
01/31/23 16:47 1,048,576 0 A --- Swapper.dat
Then it is not valid. That value for me means that an unpleasant value is introduced, and one is created by default.
I would wait created a next 500mb File

Now I will explain my frequency theory.
I think this is correct, but I don't see especially good results. Maybe the swapper is canceled partially.

It is a frequency, when you multiply a number * x frequency = n frequency
The OS/2 frequency constant is 31.25
The value has to be an entire number, that is, without decimals.
The swapper.dat if it is a file and has a size.
Examples below.

The minimum theoretical size would be 31.25*4 = 125, to have an entire number,
 the maximum depends on the space for the swapper.dat file

Therefore 125+125 = 250+125 = 375+125 = 500 .... will be valid values, which coincide with my test.

Some examples, the size of the swappath and the generated file:
swappath=L:\ 0 506000    ----31.25*16192-- = ---125*4048---
28/01/23 22:49    519.045.120      0 a---  SWAPPER.DAT
        3 file(s) 519.045.130 bytes used
                      9.307.136 bytes free

swappath=L:\ 0 3125      -----31.25*100---------
28/01/23 19:23      4.194.304      0 a---  SWAPPER.DAT

swappath=L:\ 0 6250       -----31.25*200---------
28/01/23 19:58      7.340.032      0 a---  SWAPPER.DAT

swappath=L:\ 0 15625    -----31.25*500-----------
28/01/23 20:12     16.777.216      0 a---  SWAPPER.DAT

swappath=L:\ 0 20625    -----31.25*660------------
28/01/23 20:46     22.020.096      0 a---  SWAPPER.DAT

swappath=L:\ 0 31125    ----31.25*996--------------
28/01/23 21:48     32.505.856      0 a---  SWAPPER.DAT

If you find a number that fulfills my 125xn theory that fails, my theory would be wrong.

I have been with 0 4096 for many months, with very satisfactory results, so I am with you that the 4K can be important.
Saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on January 31, 2023, 07:58:15 pm
REMY
Thanks for the virtualaddresslimit, proposed as possible.
I have only tried 3008, for my screen problem. And it doesn't work, the screen is corrupted.
In principle, the rest, and stay with those of a lifetime, proposed by Dave.

The Swappph, SwaPPath = m: \ 131072 524288, it seems that it works and is correct, but it is slow.
Generates this swapper.dat:
01/31/23 16:47 1,048,576 0 A --- Swapper.dat
Then it is not valid. That value for me means that an unpleasant value is introduced, and one is created by default.
I would wait created a next 500mb File

Now I will explain my frequency theory.
I think this is correct, but I don't see especially good results. Maybe the swapper is canceled partially.

It is a frequency, when you multiply a number * x frequency = n frequency
The OS/2 frequency constant is 31.25
The value has to be an entire number, that is, without decimals.
The swapper.dat if it is a file and has a size.
Examples below.

The minimum theoretical size would be 31.25*4 = 125, to have an entire number,
 the maximum depends on the space for the swapper.dat file

Therefore 125+125 = 250+125 = 375+125 = 500 .... will be valid values, which coincide with my test.

Some examples, the size of the swappath and the generated file:
swappath=L:\ 0 506000    ----31.25*16192-- = ---125*4048---
28/01/23 22:49    519.045.120      0 a---  SWAPPER.DAT
        3 file(s) 519.045.130 bytes used
                      9.307.136 bytes free

swappath=L:\ 0 3125      -----31.25*100---------
28/01/23 19:23      4.194.304      0 a---  SWAPPER.DAT

swappath=L:\ 0 6250       -----31.25*200---------
28/01/23 19:58      7.340.032      0 a---  SWAPPER.DAT

swappath=L:\ 0 15625    -----31.25*500-----------
28/01/23 20:12     16.777.216      0 a---  SWAPPER.DAT

swappath=L:\ 0 20625    -----31.25*660------------
28/01/23 20:46     22.020.096      0 a---  SWAPPER.DAT

swappath=L:\ 0 31125    ----31.25*996--------------
28/01/23 21:48     32.505.856      0 a---  SWAPPER.DAT

If you find a number that fulfills my 125xn theory that fails, my theory would be wrong.

I have been with 0 4096 for many months, with very satisfactory results, so I am with you that the 4K can be important.
Saludos

About your screen issue, do you have a dedicated video card or is the full storage shared with the video (HW).
May be you sould check your bios and specifically the amount of MiB allocated for the video. OS/2 needs less storage (2D rendering) and 16M or if you have other OSes installed, 32MiB is good (I have it set to 32MiB and use win7 as well ArcaOs
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on February 01, 2023, 03:35:59 pm
Hi Remy
This is a laptoop, and I have no option of any kind of configuring video in bios.

But do it about 4096 if I can tell you a curious case:
The Standard Diskcache = D, LW
The coherent would be diskcache = 4096, lw or 2048
The configured diskcache = 4096000, LW

Any number greater than 4096000 did not work for me, any smaller number did not work either.
Copies to Fat32 from computer to USB Stick, accelerate by 4 or 5 times in less time.
(Sure of the numbers that I provide)

And if everything fails you already know, call the system administrator.
Saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on February 02, 2023, 06:18:34 pm

The configured diskcache = 4096000, LW

I answer myself because I keep asking questions:
4096000/125 = 32768 Surprise is an integer.
And it won't it be a frequency in diskcache?
And I have tried the FAT32 and I changed 2048 for 2000 and
The IFS of HPFS 2048 for 2000, and new surprise I can copy from the USB stick to the hard drive, with FC2.40 and FC2.50, the arcaos-5.0.7 in four minutes, instead of a long hour, which came out With the Laser Commander.
FC2.40 and FC2.50 are closed at the end of the copy, it requires being attentive to see the final result, because it does not calculate the times well, and it appears as if the copy would be slower, but it ends it correct and rapid.
Before I could only copy the USB stick from the computer, but now I can also do it on the contrary.
The copied file for this test is:
14/01/23 19:15    1.515.440 K      0 a---  ArcaOS-5.0.7.iso
With the IFS of the JFS, I do not dare to experiment, that I leave it to you.

Saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on February 06, 2023, 10:58:22 am
With reference to Fat32 Cache, the information provided was incomplete, how strange no one is tested
This would be the right thing:
REM IFS=C:\OS2\FAT32.IFS /cache:2048 /h /q /ac:* /largefiles
IFS=C:\OS2\FAT32.IFS /cache:4000 /q /ac:* /largefiles
REM CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S

If the second cache is not canceled, the computer does not start, with a trap in Fat32, or similar failure.
Another question may be that program is the most appropriate to make copies.

Also where I say:
AFTER  ramdisk 512mb and and modify the config.sys changing the swapper to that newly created partition
I should say:
AFTER  ramdisk 512mb (The swappath gives the same where you put it)

saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Andi B. on February 06, 2023, 01:35:09 pm
Code: [Select]
[quote author=roberto link=topic=3232.msg38169#msg38169 date=1675677502]
With reference to Fat32 Cache, the information provided was incomplete, how strange no one is tested
Don't understand what you mean. Do you think someone should try to reproduce your measurements? Then maybe one problem is you're hard to understand. At least I can't follow what you're saying most of the time.

Quote
This would be the right thing:
REM IFS=C:\OS2\FAT32.IFS /cache:2048 /h /q /ac:* /largefiles
IFS=C:\OS2\FAT32.IFS /cache:4000 /q /ac:* /largefiles
REM CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S

If the second cache is not canceled, the computer does not start, with a trap in Fat32, or similar failure.
What is 'the second cache'? What should be 'canceled'? Do you mean you have to rem out the call to cachef32.exe otherwise it traps?

'with a trap in Fat32, or similar failure' - what? Trap or not? At bootup or later? Which trap? Or similar?

Quote
Another question may be that program is the most appropriate to make copies.
No clue what you want to say with this. As there is no question mark it seems you make a statement here which I don't understand.

Quote
Also where I say: ......
Sorry, maybe my English is to bad but most of the time I can't follow what you're trying to say.

Question - do you have a setting for FAT32 line in config.sys which you have measured considerable faster transfer rate? Please provide old (slow) line and new (fast) config.sys line. And you measured it (both the same of course) and tell us the speed (MBytes/s or 1 file mit xxx MByte took yyy seconds). Thanks.
Title: Re: Virtual memory, a curious phenomenon
Post by: ivan on February 06, 2023, 06:54:52 pm
Hi Andi,

I'm English and like you I don't understand what he is rabbiting on about.  I also don't see what he is trying to gain with all the messing about he does after all 'if it works why change anything' is the rule I worked under before my retirement, nearly 30 years ago, and still do today.

I would be nice if he explained what he is trying the achieve but that seems to be some nebulous goal without any real reasoning.
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on February 06, 2023, 09:07:33 pm
It is about being able to make a copy from a USB Stick, from a 1.5GB file to the computer, in reasonable spread, between 7 to 12 minutes, with the standard configuration takes 60 to 65 minutes.
But not only that, it means that my frequency theory for CACHE, swappath, works, in a visible way.

Your standard config.sys 
Quote
IFS=C:\OS2\FAT32.IFS /cache:2048 /h /q /ac:* /largefiles
CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S

swappath=C:\os2\system\ 2048 2048
My experimental proposal
Quote
IFS=C:\OS2\FAT32.IFS /cache:2000 /h /q /ac:* /largefiles
rem CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S

swappath=L:\ 2000 409600

Provisted screen in Memory ON (I don't know if it will be necessary or not.)
Quote
If the second cache is not canceled, the computer does not start, with a trap in Fat32, or similar failure.
What is 'the second cache'?
cachef32.exe

Quote
What should be 'canceled'? Do you mean you have to rem out the call to cachef32.exe otherwise it traps?
yes, trap with fat32 or On another computer I showed me a complete black screen and stopped the start.


Quote
Quote
Another question may be that program is the most appropriate to make copies.
No clue what you want to say with this. As there is no question mark it seems you make a statement here which I don't understand.
I mean FC2 2.40 closes when the copy ends, 7 to 12 minutes, although it reflects times similar to the standard, while copying 60 minutes, but ends it before.
Other possible programs would be Copy, Xcopy, Laser Commander WPS etc ...

Quote
Question - do you have a setting for FAT32 line in config.sys which you have measured considerable faster transfer rate? Please provide old (slow) line and new (fast) config.sys line. And you measured it (both the same of course) and tell us the speed (MBytes/s or 1 file mit xxx MByte took yyy seconds). Thanks.
The file is
14/01/23 19:15    1.515.440 K      0 a---  ArcaOS-5.0.7.iso
The copy from pc to usb stick is in 4 minutes with the disk cache proposed before.
With this proposal is the opposite, copying it from USB stick to HD
saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: ivan on February 07, 2023, 06:27:00 am
Hi roberto,

Without knowing what USB stick you are using there can not be any real comments made - not all USB sticks are made equal, some are better quality than others.  I have no problems copying a 2.7 GB ISO file to or from a USB stick in less than 2 minutes on one stick but it takes 4 minutes on another.

When you do tests you should give all the information about the test, make and model USB stick, imf you tried the same test with another brand of USB stick, What is the hardware for the USB- is it native mode or a pass through from a USB v3 controller, and so on.  All those things make a bigger difference to copy speed than messing about with the config.sys settings.
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on February 07, 2023, 02:33:34 pm
Hi roberto,

Without knowing what USB stick you are using there can not be any real comments made - not all USB sticks are made equal, some are better quality than others.  I have no problems copying a 2.7 GB ISO file to or from a USB stick in less than 2 minutes on one stick but it takes 4 minutes on another.

When you do tests you should give all the information about the test, make and model USB stick, imf you tried the same test with another brand of USB stick, What is the hardware for the USB- is it native mode or a pass through from a USB v3 controller, and so on.  All those things make a bigger difference to copy speed than messing about with the config.sys settings.

Like Roberto,

Using usb sticks (USB2 or USB3 or above high speed up to above 300BiB/s  (up to 420MiB under win10)
But not only, writting to FAT32 partition on a disk takes long, very long with a speed less than 1MiB !!! (nearly 300TiB fat32 partition)
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on February 07, 2023, 03:54:18 pm
Hi Ivan
Without knowing what USB stick you are using there can not be any real comments made - .
The port is a USB 3.0, with a 32GB micro SD, from the Kingston brand, EHCI.
But I have done it with the same SD and the same port, just changing the config.Sys the file to copy the same.

saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: ivan on February 08, 2023, 08:00:00 pm
As far as I know that SD card requires an SD to USB adapter to allow it to be used as a USB device.  There is your first problem - those adapters are not the fastest way of reading SD cards via USB.

Try again with a real USB stick, something like a Philips 32 Gb unit and see what your timing is then.  As an aside, the USB chipset of the computer also has some input to the speed of inserted USB sticks.
Title: Re: Virtual memory, a curious phenomenon
Post by: Andi B. on February 08, 2023, 08:12:15 pm
I only would test with an USB3 harddisk. Fastest are the 3,5" ones. But for FAT32 test every disk should be fast enough. Sticks are so different and sometimes transfer rate starts with high speed but drops afterwards -> unusable for comparison tests.
Title: Re: Virtual memory, a curious phenomenon
Post by: Dave Yeo on February 08, 2023, 11:59:55 pm
As far as I know that SD card requires an SD to USB adapter to allow it to be used as a USB device.  There is your first problem - those adapters are not the fastest way of reading SD cards via USB.

Try again with a real USB stick, something like a Philips 32 Gb unit and see what your timing is then.  As an aside, the USB chipset of the computer also has some input to the speed of inserted USB sticks.

My fastest USB device, ignoring the HD, was a SD card using a converter.
Most of these USB sticks seem like crap, super slow with FAT32 and not faster with JFS.
Title: Re: Virtual memory, a curious phenomenon
Post by: Dave Yeo on February 09, 2023, 12:09:08 am
With reference to Fat32 Cache, the information provided was incomplete, how strange no one is tested
This would be the right thing:
REM IFS=C:\OS2\FAT32.IFS /cache:2048 /h /q /ac:* /largefiles
IFS=C:\OS2\FAT32.IFS /cache:4000 /q /ac:* /largefiles
REM CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S

If the second cache is not canceled, the computer does not start, with a trap in Fat32, or similar failure.
Another question may be that program is the most appropriate to make copies.

Also where I say:
AFTER  ramdisk 512mb and and modify the config.sys changing the swapper to that newly created partition
I should say:
AFTER  ramdisk 512mb (The swappath gives the same where you put it)

saludos

Did you think to check what CACHEF32 does? Along with caching, it also does the unicode translation. Remove it and create a file with umlauts or such, put it back and you get crashes, same if you shared the stick, might even crash Windows, though they probably protect against it.
What you might want to do is turn off lazy writing, which some say really helps with USB sticks and the ramdisk. From \sys\book\fat32.inf, under CACHEF32,
Code: [Select]
/L:ON|OFF Sets lazy writing ON or OFF (default is ON).
And also from the same inf, under System-critical files and booting,
Code: [Select]
It is not possible to place the OS/2 swapfile (SWAPPER.DAT) on a FAT32 volume.
So it seems that you are using the default swap file in \os2
HPFS would work for the swap file on the ram disk, JFS maybe with some screwing around if the ram disk is preserved during a warm boot, one boot to format the disk JFS and a reboot to put the swap file there.
Edit: You should backup your FAT32 volumes, format and with CACHEF32 installed, restore. The same sort of problem happens with JFS if unicode.sys goes missing, ask Remy.
Edit2:, also the max size of the cache is 2048 KB, so your /cache:4000 likely just loads the default, 1024KB
Also remove the /Q when testing so you see any messages

Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on February 09, 2023, 01:17:42 am
I only would test with an USB3 harddisk. Fastest are the 3,5" ones. But for FAT32 test every disk should be fast enough. Sticks are so different and sometimes transfer rate starts with high speed but drops afterwards -> unusable for comparison tests.

Hi Andy, read my comments
With SATA fat32 HD partition, it is very slow too (near 1MB/s ... in write)

It looks like it is partition size dependant ! may it be possible ?
(A small USB key is faster than the SATA FAT32 HD !)
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on February 09, 2023, 04:15:34 pm
Sorry Dave, this is in mistake. Keep in mind that I do not have a manual, and I do evidence and error, and sometimes I take bad conclusions that then correct.

Quote
With reference to Fat32 Cache, the information provided was incomplete, how strange no one is tested
This would be the right thing:
REM IFS=C:\OS2\FAT32.IFS /cache:2048 /h /q /ac:* /largefiles
IFS=C:\OS2\FAT32.IFS /cache:4000 /q /ac:* /largefiles
REM CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S
The correct is:
IFS=C:\OS2\HPFS.IFS /CACHE:2000 /CRECL:4 /AUTOCHECK:GL*
IFS=C:\OS2\FAT32.IFS /cache:2000 /h /q /ac:* /largefiles
rem CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S

swappath=L:\ 2000 409600

REM DISKCACHE=D,LW
DISKCACHE=4096000,LW

Provisted screen in Memory ON (I don't know if it will be necessary or not.)

Quote
Along with caching, it also does the unicode translation. Remove it and create a file with umlauts or such, put it back and you get crashes, same if you shared the stick, might even crash Windows, though they probably protect against it.
What you might want to do is turn off lazy writing, which some say really helps with USB sticks and the ramdisk. From \sys\book\fat32.inf, under CACHEF32,
Well, I have tried it in Win10, and no problem.
I have been able to copy the files without problem, to win10.
I have a ramdisk 512mb in L:\ with the swappath here.

I have tried without /q and it shows me nothing strange in the start, but it does not work at high speed, it only works 60 minutes, slow.

-Ivan
Thanks for your comment, but I have to use these micro SD, for camera photos, and other equipment.
saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: ivan on February 10, 2023, 04:22:21 pm
What camera do you have?  I do not remove the cards from my Nikon D60 or D3200 cameras, just plug in the correct USB cable and use  Cameraderie v1.5.3   (C) Copyright 2006, 2007  R L Walsh to copy the photos to my hard disk - very quick, simple and easy to do.

Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on February 10, 2023, 09:35:54 pm
Sorry Dave, this is in mistake. Keep in mind that I do not have a manual, and I do evidence and error, and sometimes I take bad conclusions that then correct.

The correct is:
IFS=C:\OS2\HPFS.IFS /CACHE:2000 /CRECL:4 /AUTOCHECK:GL*
IFS=C:\OS2\FAT32.IFS /cache:2000 /h /q /ac:* /largefiles
rem CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S

swappath=L:\ 2000 409600

REM DISKCACHE=D,LW
DISKCACHE=4096000,LW

Provisted screen in Memory ON (I don't know if it will be necessary or not.)


I just did a test changing the 2 lines as Roberto did (cache at 2000 and rem the call).
What I can say is Whouah ! how fast it is on fat32 !  (My HDD on SATA is very fast compared to before where it took long seconds copy a few megabytes.
Title: Re: Virtual memory, a curious phenomenon
Post by: Dave Yeo on February 10, 2023, 10:50:15 pm
Did you try simply turning off lazywrite? The manual is in x:\sys\book where X: is your boot drive. As mentioned, cachef32 does the unicode conversion.
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on February 11, 2023, 07:21:46 pm
Did you try simply turning off lazywrite? The manual is in x:\sys\book where X: is your boot drive. As mentioned, cachef32 does the unicode conversion.
I have already tried it with the /L: OFF, and copying from the PC to the USB Stick is 3 seconds faster.
But copying from the USB Stick to the PC is 61 minutes, against 7 to 12 minutes.
That difference is a long time.
saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on February 15, 2023, 05:33:48 pm
Did you try simply turning off lazywrite? The manual is in x:\sys\book where X: is your boot drive. As mentioned, cachef32 does the unicode conversion.
I have already tried it with the /L: OFF, and copying from the PC to the USB Stick is 3 seconds faster.
But copying from the USB Stick to the PC is 61 minutes, against 7 to 12 minutes.
That difference is a long time.
saludos

Hi Roberto,

On  your IFS, use a multiple of 64K for the cache value instead of 4k
|  IFS=C:\OS2\FAT32.IFS /cache:2000 /h /q /ac:* /largefiles                                      << 2000 not good
|  rem CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S

As multiple of 64k, use 1984 instead of 2000 and rem out the Call= which will not trap.
IFS=C:\OS2\FAT32.IFS /cache:1984 /h /q /ac:* /largefiles
CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:5000 /S
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on February 15, 2023, 08:26:02 pm
Hi Remy
I have tried it and slows me 65 minutes, copying from USB stick to HD
saludos
Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on February 16, 2023, 01:09:14 am
It only use of the next CALL statement.

I had dounbt about my previous tests due I had several processes running in background...
Now, I redo several test in read and write from/to SSD sata partitions (one into fat32, the other as jfs)

I used a very big file and stopped the process after 60s (may be 1/4s differences but as precise as possible)

After 60s (CTRL C) from SATA SSD (fat32) to SATA SSD (JFS)
With -> IFS=C:\OS2\FAT32.IFS /cache:1984 /h /q /ac:* /largefiles
With -> REM CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:4000 /S /L:ON
15/02/23 23:55    369 011 200      0 a---  dsd64-2ch (3).zip

With -> IFS=C:\OS2\FAT32.IFS /cache:1984 /h /q /ac:* /largefiles
With -> CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:4000 /S /L:OFF
16/02/23  0:03    567 854 592      0 a---  dsd64-2ch (3).zip

With -> IFS=C:\OS2\FAT32.IFS /cache:2048 /h /q /ac:* /largefiles
With -> CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:4000 /S /L:OFF
16/02/23  0:13    566 098 944      0 a---  dsd64-2ch (3).zip

With -> IFS=C:\OS2\FAT32.IFS /cache:2048 /h /q /ac:* /largefiles
With -> CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:4000 /S /L:ON
16/02/23  0:19    566 619 136      0 a---  dsd64-2ch (3).zip

With -> IFS=C:\OS2\FAT32.IFS /cache:2000 /h /q /ac:* /largefiles
With -> REM CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:4000 /S /L:ON
16/02/23  0:26    370 636 800      0 a---  dsd64-2ch (3).zip


After 60s (CTRL C) from SATA SSD (JFS) to SATA SSD (FAT32)
With -> IFS=C:\OS2\FAT32.IFS /cache:2000 /h /q /ac:* /largefiles
With -> REM CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:4000 /S /L:ON
16/02/23  0:36    478 121 472      0 a---  dsd64-2ch (3).zip

After 60s (CTRL C) from SATA SSD (JFS) to SATA SSD (FAT32)
With -> IFS=C:\OS2\FAT32.IFS /cache:2048 /h /q /ac:* /largefiles
With -> CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:4000 /S /L:OFF
16/02/23  0:43    426 882 560      0 a---  dsd64-2ch (3).zip

After 60s (CTRL C) from SATA SSD (JFS) to SATA SSD (FAT32)
With -> IFS=C:\OS2\FAT32.IFS /cache:2048 /h /q /ac:* /largefiles
With -> CALL=C:\OS2\CACHEF32.EXE /f /p:2 /m:50000 /b:250 /d:4000 /S /L:ON
16/02/23  0:50    282 464 256      0 a---  dsd64-2ch (3).zip

In this kind of test, the L:OFF has a big impact but overall, the speed is very very slow for SSD's !
testing from JFS to JFS for one very big file results into nearly 98MB/s which is of course slow for an SSD able to speed up to 520M !
(Disk aligment may probably impact - isn't correctly done for SSD and may be trim need [not done since a few weeks])

I didn't test usb keys yest and I know it needs more time because as soon it warms up, speed will dramatically slowdown
I have a 1TB usb key with a speed nearly 500MB. Under windows, It will start nearly at this speed but after a few seconds, the speed start to drop up to a slow value about 6M ! (the key been warm and into speed regulation to protect it)   
     
Title: Re: Virtual memory, a curious phenomenon
Post by: roberto on February 16, 2023, 10:47:01 pm
I used a very big file and stopped the process after 60s (may be 1/4s differences but as precise as possible)
How do you do the copy and cut it in a time? I have tried it with a zip, and when I cut it I have not copied only one part of the file.

Quote
After 60s (CTRL C) from SATA SSD (fat32) to SATA SSD (JFS)
Your test is more fast fat32 to jfs
Quote
After 60s (CTRL C) from SATA SSD (JFS) to SATA SSD (FAT32)
Your test is more slow jfs to fat32
With me with the USB stick, it happens to me, it is slower fat32 to jfs.
I am curious to do some proof as you comment.
Saludos

Title: Re: Virtual memory, a curious phenomenon
Post by: Remy on February 16, 2023, 11:36:08 pm
I used a very big file and stopped the process after 60s (may be 1/4s differences but as precise as possible)
How do you do the copy and cut it in a time? I have tried it with a zip, and when I cut it I have not copied only one part of the file.

Quote
After 60s (CTRL C) from SATA SSD (fat32) to SATA SSD (JFS)
Your test is more fast fat32 to jfs
Quote
After 60s (CTRL C) from SATA SSD (JFS) to SATA SSD (FAT32)
Your test is more slow jfs to fat32
With me with the USB stick, it happens to me, it is slower fat32 to jfs.
I am curious to do some proof as you comment.
Saludos

The zip file has a size near 4GB.
Positionning into the right directory on both side.
Then, from/to issued prompt cmd :  copy filename to w:   or reverse  copy filename to y:
To stop, issued CTRL/C

The problem using a prompt cmd window is that messages are impacted by the video refresh, using one big file prevent copied message to be issued until the end of the copy.

As it is seen, the L:OFF / ON has a big impact on the write (it is for this) on FAT32

Copying files to or from my local network NAS is faster !