QSINIT updated - now with HPFS support for the ramdisk

[rbri]

There is a new version available that supports hpfs-formated ramdisk.
Works without problems here.

Many thanks!

[Michaelhz]

This is really great news. We can easily put swapper.dat on the ramdisk now. This is a great step forward.

Works perfect here.

Many thanks to Dmitry Zavalskov!

Michael

[Doug Bissett]

This is really great news. We can easily put swapper.dat on the ramdisk now. This is a great step forward.

Works perfect here.

Many thanks to Dmitry Zavalskov!

Michael

Many thanks indeed.

But why put the swap file on a RAMDISK? If you have enough memory to use a RAMDISK, you probably don't use the swap file anyway (I have never seen the swap file used when I have more than about 1.5 GB of memory). If you do need the swap file, you would be better off to not use a RAMDISK, and let the system use the memory. Putting %TEMP% on the RAMDISK makes sense.

Don't forget that running out of swap space can be deadly. It is best to have the swap file on the disk with the most free space, if you actually need a swap file.

I applaud the step forward, to use HPFS, but I also encourage one more step, to use JFS. I quit using the HPFS driver a few years ago (when it became possible to boot from JFS). The main reason was that it saves a good chunk of very scarce shared memory space. JFS also works better than HPFS (IMO). To use HPFS on the RAMDISK, I need to install the HPFS driver. That is not going to happen on my systems.

[dbanet]

Doug, the author of QSINIT has reported that the support of formatting into JFS will not be introduced in QSINIT as being too hard and senseless, as there is no need for a journaling file system on a RAM drive.

Doug, what Michael meant by really great news was the ability to actually swap out to the physical memory, directly inaddressable by OS2KRNL, which theoretically means the ability to address more than four gibibytes of virtual memory in system.

Michael, that could be very good news, but unfortunately there is actually one another drawback in any existing OS2KRNL, no matter released by David or the Phoenix OS/4 project: it appears (please check out my signature) that there is a technical limitation of all memory allocated by all processes, i.e. swapped + not swapped memory. I've once run a test with 14.104a_SMP and the kernel being the latest released by Phoenix OS/4 back at that time: I've checked that swapping is turned on on my system, and that the drive where the SWAPPER.DAT file is located is big enough, and tried to allocate as much memory, as I could. I was spawning processes each allocating some fixed amount of memory, which I selected, but no matter what amount of memory each process allocated, its sum had never been greater than 4 GiB, because every time eventually I either got a system hang, or a malloc failure, or both. This was happening on both kernels.

I've said this is a technical limitation, because it actually seems to be just a single drawback in the current OS2KRNL implementation, as there is no theoretical limitation of the overall amount of memory allocated by all processes. I can't be sure (still please take my signature into consideration), but it seems to be a limit that is not that hard to overcome (means, way easier than to introduce direct PAE support in OS2KRNL). I really suggest the kernel developers to cooperate, share the code, and solve the actual problems...

Regards.

[Doug Bissett]

Doug, the author of QSINIT has reported that the support of formatting into JFS will not be introduced in QSINIT as being too hard and senseless, as there is no need for a journaling file system on a RAM drive.

Well, yes there is a need. If I load the HPFS driver, I constantly run into "insufficient memory" problems. Without it, I don't. The RAMDISK needs a file system that is better than FAT(32). I format the FAT32 drive to JFS during boot, and it works well. The tricky part is, that the file system somehow thinks that JFS does not properly support long file names, until I unmount the drive, and mount it again, after the GUI gets started (the procedure is somewhere on OS2WORLD).

Doug, what Michael meant by really great news was the ability to actually swap out to the physical memory, directly inaddressable by OS2KRNL, which theoretically means the ability to address more than four gibibytes of virtual memory in system.

Hmmm. You read something into that, that I don't see. You cannot use more than 4 GB of memory directly. You could use the "extended" memory by paging it into/outof "main" memory. This is a totally different thing than using the swap file, and the kernel limit of 4 GB still exists. That doesn't mean that a program is actually limited by that, IF it knows how to use the"extended" memory. As I understand it, the RAMDISK support provides a framework for using "extended" memory, but there is no program interface for that, yet. A program would also need to be specifically written to use the "extended" memory (as the RAMDISK is). With some serious rework in the memory allocation area, it may be possible to dynamically use the "extended" memory, but I doubt if anybody has the knowledge to be able to do it (I would love to be proven wrong).

So, the kernel knows about private memory, and shared memory, for a total of 4 GB. Programs could be taught to use "extended" memory, that the kernel knows nothing about. That could be a little tricky if the system crashes, and somebody is trying to figure out a system dump, but that should not be a killer. I see "extended" memory being used exclusively for program data, so it doesn't matter if the kernel knows about it, or not. DOS has used a similar memory setup, for years, and DOS still only knows about 1 meg.

[Dave Yeo]

What Boris means is that you should be able to have 3 programs each allocating 2 GBs of memory each. The 4GB (actually closer to 3.5GB as the upper is used by BIOS, PCI etc) is a per process limitation and IIRC the i486 can keep track of something like a TB of virtual memory.
Another limitation is that the swap file itself is limited to 2GBs, might be patchable to 4Gbs.

[Doug Bissett]

What Boris means is that you should be able to have 3 programs each allocating 2 GBs of memory each.

Well, that is likely to never happen, unless the OS/4 project makes some huge strides (unlikely, from past performance). You have 4 GB (okay, 3.5 GB) to work with, and that is the limit. However, if each program used 1 GB of "extended" memory, it would be possible to do as suggested.

IIRC the i486 can keep track of something like a TB of virtual memory

I doubt if they can, but even if they can, OS/2 cannot. It is designed to work with an addressing scheme that is limited to 3.5 GB. Whatever is done to add more memory access must either work outside of what the kernel knows about (like the RAMDISK) or, the kernel requires major modifications. I know which one is most likely to happen.

Another limitation is that the swap file itself is limited to 2GBs, might be patchable to 4Gbs.

I suspect that there isn't much point. You would need to page out ALL of memory to use it. If memory is available, it would not be paged out. Either that, or there wouldn't be enough real memory for the system to run in. I believe that the term was "thrashing" to death. The swap file is just an extension of real memory, and that still has the 3.5 GB limit.

The "extended" memory would need to be used more like a high speed, direct access, temp file, bypassing the file system. Today, the RAMDISK can be used in that way, but the temp files need to be accessed through the file system (including caching), which adds some overhead.

From what I have seen, there are not many modern programs that make much use of TEMP files anyway, and those files tend to be small, so there is limited benefit from that approach, today.

We have a new toy, but we need to figure out how to use it effectively.

[Dave Yeo]

You're misunderstanding the difference between address space and virtual address space. Just with segments there is actually 256 segments of 4GB each available and OS/2 understands this.
The hardware itself allows a large virtual address space and has the means to keep track of it and the OS supports this to an extent. Of course OS/2 is old and they didn't put much work into, what at the time was considered insane memory usage but the capability is there. 32 bit NT as an example can easily use swap files that are larger then 4 GBs, eg see http://en.wikipedia.org/wiki/Paging#Windows_NT where they casually mention creating 6GB and 16 GB swap files.
Another example is OS/2 1.x where the directly addressable memory on a 286 was 16 MB yet the total virtual memory limit was 1GB which is where our low memory limit comes from (half for apps and half for the OS).
We should be able to, amongst multiple programs, allocate about half a TB of memory with the other half given to hardware and OS overhead. (Probably less due to limited amount of page table entries, but that can be worked around by using 1 (4?) MB pages instead of 4KB pages)

[Doug Bissett]

You're misunderstanding the difference between address space and virtual address space. Just with segments there is actually 256 segments of 4GB each available and OS/2 understands this.

That is what the hardware is capable of doing. OS/2 doesn't use it that way, and I doubt if it "knows how", or somebody would have figured it out, by now.

Of course OS/2 is old and they didn't put much work into, what at the time was considered insane memory usage but the capability is there.

In fact it is still insane memory usage. Unfortunately, there are more insane programmers around.   

Probably less due to limited amount of page table entries, but that can be worked around by using 1 (4?) MB pages instead of 4KB pages

If somebody can figure out the page handling system, the problem could likely be solved. The kernel would still be limited to 3.5 GB addressing, but that 3.5 GB could be a different 3.5 GB space than the next program is using (not quite that simple, I am sure, but...). In fact, I got the impression that that is exactly what PAE addressing does. The problem is with managing it, and telling the kernel that programs can actually have overlapping addresses (in different address spaces). I think it would be much easier to simply work outside of the kernel, for private data storage, but the interface needs to be defined, and programmed. If that is done, the shared memory space can be expanded into the area vacated by the private memory. Not a "perfect" solution, but it should be possible, without trying to mess with the kernel, for somebody who understands the concept. Then, programmers would need to change their memory calls to access it that way.