OS2 World Community Forum

WebSite Information => Article Discussions => Topic started by: Martin Iturbide on December 10, 2020, 03:56:46 pm

Title: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Martin Iturbide on December 10, 2020, 03:56:46 pm
And yes, in my dreams I still think about a 64 bit recompiled OS/2 kernel :)

I would love to theorize about that in another forum thread. I know it is too hard, but it will be interesting to brainstorm with people with more knowledge on this subject. I will look on other past threads about this topic.

Regards
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Joop on December 10, 2020, 10:17:08 pm
As far as I know is that the memory is a little less than 4Gb. But this is for the whole OS, no matter if you have 1 kernel or 4 or 8 or... What could be done is each kernel its own little less than 4GB. So with a 4 kernel system you could install 16Gb memory. And 64-bit is a hoax. I have a W10 64-bit from work, everything is bigger including what you make with the applications, but no improvement on the 32-bit applications in my OS/2 system. We complain sometimes about this and that, with Windows 10 its worse. Each day I encounter problems with USB. The Wacom is sometimes working and sometimes not. Now this can easy be fixed, but its not okay. Office from Microsoft is a mess, really, no application does have the same file interface, parts are deaf for the cloud, some programs are even worse then found in the shareware corner. And you don't want to know what the company has to pay for this package.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Lars on December 10, 2020, 11:22:46 pm
Physical addresses under OS/2 are also limited to 32-bit. Which means your approach of "4 GB of physical memory per core" cannot work.
You cannot address beyond the physical 4 GB boundary with the current OS/2. You would have to completely reimplement OS/2 memory management to support 64-bit physical addresses (and linear addresses).
Which becomes a massive problem because many new systems "waste" physical address space below the 4GB boundary by simply leaving it unused or by placing large regions of "non system-memory", for example, PCI address space.
And that's why there are machines that have 4 GB of system memory but can only use 2.5 GB or so under OS/2. That's because the other 1.5 GB of system memory are physically addressable beyond the 4 GB physical address boundary.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Jochen Schäfer on December 11, 2020, 09:42:36 am
And 64-bit is a hoax. I have a W10 64-bit from work, everything is bigger [..]
That's not a hoax, it is to be expected. If you want to address 64bit instead of 32bit, you need a bigger pointer. Most C/C++ compilers will make their types bigger, e.g. int going from 32bit to 64bit, so the data grows.  Everything grows, if you switch word size (32bit vs. 64bit). It was like that with the translation from 8bit to 16bit, then from 16bit to 32bit.
So, in a nutshell: Making your word size bigger, makes your executables and your data bigger. That's the downside for getting a so much bigger address space.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Jochen Schäfer on December 11, 2020, 09:46:56 am
Physical addresses under OS/2 are also limited to 32-bit. Which means your approach of "4 GB of physical memory per core" cannot work.
I remember, reading that the 64bit Darwin kernel always presents 32bit software with a 4 GB address space of its own. It does this via virtual addressing. This could be a way to go for OS/2, I think.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Lars on December 11, 2020, 10:15:05 am
Ok,that refers to the logical/linear address (the one the CPU translates to the physical address via the paging mechanism). That could remain 32-bit.
Still, your memory management has to support 64-bit physical addresses. The memory is a global resource, a physical address is unique, in contrast to a logical one.
And this means a rewrite of the memory management.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Martin Iturbide on December 11, 2020, 02:52:11 pm
Hi

I slip the topic, since it is a very interesting discussion.

Regards
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Martin Iturbide on December 11, 2020, 03:21:37 pm
Hi

Since the OS/2 kernel is not open source, I always thought that some alternative for OS/2 kernel evolution issue will be to replace completely the OS/2 kernel for a different 64bit kernel and interpret the OS/2 personality with the IBM binaries over it. People will complain that the kernel is the OS, but I'm starting to believe that OS/2 is more about the experience of the desktop and being able to run the OS/2 applications software that we like and use.

Maybe like Project 2ine (https://www.osnews.com/story/30205/project-2ine-os2-binaries-on-linux/) tried to do, but not re-implementing all the OS/2 API (like PM) in one shot, but trying to use first the IBM Binaries tricking them to think they are running on the OS/2 kernel while not. 2ine tried to do a Wine for OS/2 under Linux, maybe we should bring the OS/2 components over a different kernel instead. Of course, all OS/2 drivers and application that uses a driver (like some utilities) will turn useless.

I just trying to think how something like that should look from a architectural high level point view.

Regards
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Roderick Klein on December 11, 2020, 09:46:09 pm
Physical addresses under OS/2 are also limited to 32-bit. Which means your approach of "4 GB of physical memory per core" cannot work.
You cannot address beyond the physical 4 GB boundary with the current OS/2. You would have to completely reimplement OS/2 memory management to support 64-bit physical addresses (and linear addresses).
Which becomes a massive problem because many new systems "waste" physical address space below the 4GB boundary by simply leaving it unused or by placing large regions of "non system-memory", for example, PCI address space.
And that's why there are machines that have 4 GB of system memory but can only use 2.5 GB or so under OS/2. That's because the other 1.5 GB of system memory are physically addressable beyond the 4 GB physical address boundary.

It might be possible to use the ArcaOS loader to use memory in large programs above the 4 GB memory border. But these are rough idea's. It would make more sense to see if you can modify large GCC applications to use such scheme. Do not modify the kernel, modify the application. Hopefully next year we can look further at this and see if simple proof of concept can be delivered.  To be clear again this is an idea.

Roderick Klein
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Andreas Schnellbacher on December 11, 2020, 10:42:43 pm
How I understand it, its mainly a question of adding PAE to OS/2. 64 bit wouldn't be the next step, and will never be. With PAE there would exist 4 GB address space per application. 32 bit variants of many OS have that implemented.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Doug Bissett on December 12, 2020, 09:25:48 am
How I understand it, its mainly a question of adding PAE to OS/2. 64 bit wouldn't be the next step, and will never be. With PAE there would exist 4 GB address space per application. 32 bit variants of many OS have that implemented.

PAE already exists for OS/2. QSINIT uses it with the RAMDISK, and there is a program interface to use it in other ways (pretty limited, I think). The problem is, that the RAMDISK is the only application that uses it, Nobody else has stepped up to expand on it.

I agree that OS/2 will never be 64 bit. Somebody may think they can do it, but it would be a lifetime project. I doubt if it would be possible to even identify all of the things that would need to be done to make the change.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Lars on December 12, 2020, 11:57:55 am
I don't know how PAE actually works.
But what I know is that the support that QSINIT / the AOS loader offer to access memory > 4 GB physical address is effectively based on offering 512-byte sectors to use by a requester.
This is not surprising (and also pretty clever :-)), as the QSINIT supplied RAM disk driver uses that memory as file data sectors (in the real sense of the word).
However, that also means that they offer their own sector addressing API for use by other applications but there can only be ONE user at any one time which means you will then have to give up on the RAM disk support and you cannot have multiple applications using different parts of that memory at the same time.
In short, this extension is not generic byte addressable support of memory as the CPU understands it. It's not memory management as we know it and therefore applications have to be written to use it.
Rather think of it as if you were accessing external video memory on an adapter card. A special purpose for a special user.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Andreas Schnellbacher on December 12, 2020, 01:30:22 pm
PAE already exists for OS/2.
That is wrong. The question is how to make it available for the system, not for a RAM disk. Other OS do this.

What would you gain with a RAM disk, if the usual memory exhausts? Don't mix up memory space with hard disk (or RAM disk) space.

BTW: I can't comment on Lars' post.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Doug Bissett on December 12, 2020, 07:55:34 pm
Quote
That is wrong. The question is how to make it available for the system, not for a RAM disk. Other OS do this.

It is not wrong. PAE exists. The problem is, that there is no real API to use it. Some programs could use it, as it is (raw memory space), for some things, but that could be expanded, if somebody knows how to do it. The one program, that comes to mind, is CLAMD.EXE, which loads huge amounts of data into upper shared memory space (at least it is UPPER shared space). That could live in PAE memory, with a bit of work. I am sure there are other things that could use it, without having to write a whole API. Another possibility is to use PAE memory for disk cache. At least the source code for JFS, and FAT32, is available.

Perhaps some current API could be redirected to use PAE memory, rather than upper shared memory (for instance). There are possibilities, if someone knows how to do it, and is willing to try. Part of the problem is, that not all computers have more than 4 GB of physical memory, but most new computers do have more, and it could be used. Another part of the problem is that many OS/2 programmers have lost the ability to do much more than port programs from other platforms (which is actually what is causing the problem, in the first place, but is probably necessary), and they just don't know how to do things like this any more. Those who do have the knowledge are far to busy doing more important things.

Any part of the current memory space, that can be eliminated, leaves more space for other things to use. The RAMDISK works very well, but there is a limit to how much space you can actually use as a disk drive that needs to be loaded at boot time, and unloaded at shut down, if you want to use it with data that must to be saved over a boot (in some cases, the RAMDISK can still contain the contents, but there is no guarantee about that).

Don't say "you can't do that". Ask yourself "how can i do that". That sort of thinking produced the QSINIT package, and the QSINIT package has enabled more than just the RAMDISK. It also enables using fragmented memory, that is often found in new computers. It is essential to the UEFI support that is being developed (which includes better support for DOS/WinOS2), and it will probably be used for many more things as the need arises. Small steps lead to greater things.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Andreas Schnellbacher on December 12, 2020, 08:44:32 pm
Doug, just try to understand what Lars wrote.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Dave Yeo on December 12, 2020, 08:47:23 pm
Is it just me, but I find the ramdisk to be pretty slow, barely faster then my USB2 disk. Disk 4 is the Ramdisk and disk 5 is the USB3 on USB2 HD.
Code: [Select]
Hard disk 4: 255 sides, 1138 cylinders, 63 sectors per track = 8926 MB
Drive cache/bus transfer rate: 26649 k/sec
Data transfer rate on cylinder 0   : 23463 k/sec
Data transfer rate on cylinder 1136: 20314 k/sec
CPU usage by full speed disk transfers: 11%
Average data access time: Disk read error.
Multithreaded disk I/O (4 threads): 22193 k/sec, 12% CPU usage

Hard disk 5: 255 sides, 60085 cylinders, 255 sectors per track = 1907728 MB
Drive cache/bus transfer rate: 25500 k/sec
Data transfer rate on cylinder 0   : 21492 k/sec
Data transfer rate on cylinder 60083: 21287 k/sec
CPU usage by full speed disk transfers: 1%
Average data access time: 20.4 ms
Multithreaded disk I/O (4 threads): 21347 k/sec, 2% CPU usage

I had the impression that the PAE disk interface is actually implemented as a char device rather then a block device.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: OS4User on December 12, 2020, 08:51:10 pm
Just  to clarify few things.

Almost all user and system code operates with virtual memory only.

Processor in a 32 bit system can map physical memory  into the virtual memory (paging)  in 3 ways:
- "32-bit"  32 bit phys address  (4Gb max address)
- PAE       64 bit phys address
- PSE36   36 bit phys address  (64Gb max address)

OS/2 kernel  uses "32-bit" paging thus can address 4Gb of phys mem only.  To use other paging mode it is nessasery to rewrite at least component Pager in the kernel  and almost all drivers (some of them expect that  phys address  can be 32bit only).

To rewrite Pager is a really huge job.  I'm quite skeptical that it is possible on enthusiasm only.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Doug Bissett on December 13, 2020, 04:29:53 am
Quote
Doug, just try to understand what Lars wrote.

What Lars wrote is probably correct, but that doesn't mean that it can't be changed, if somebody tries. Before QSINIT was created, most people were saying "it can't be done". It was done. now get over it,and try to improve it.

Quote
Is it just me, but I find the ramdisk to be pretty slow, barely faster then my USB2 disk. Disk 4 is the Ramdisk and disk 5 is the USB3 on USB2 HD.

I find that it depends on what file system you use. FAT16 is far faster than any of the other options (but has other limits, that make it not suitable). FAT32, is the slowest, but FAT32 is always slow, even in windows (and it has other limits, that make it not suitable). I usually use HPFS, and try to turn off as much of the cache, and lazy write, options as I can. It would probably be much better if the cache could be totally eliminated (why cache something that is going to memory anyway? It is just unnecessary overhead). I have used JFS on the RAMDISK, but then it must use the JFS cache, which actually slows it down, and you need the cache for other disks, so you can't just turn it off.

Quote
To rewrite Pager is a really huge job.  I'm quite skeptical that it is possible on enthusiasm only.

Who said anything about "Pager"? Pager is hardly ever used, when the system has more than 2 GB of real memory. That can also be put on the RAMDISK (using HPFS), but if you have 3.5 GB of real memory, available to OS/2, Pager will never be used anyway. What we need, is to offload the shared memory stuff, somehow. Private memory could also use it, if there is an access method (perhaps that would be easier).
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: OS4User on December 13, 2020, 10:27:22 am

Quote
To rewrite Pager is a really huge job.  I'm quite skeptical that it is possible on enthusiasm only.

Who said anything about "Pager"? Pager is hardly ever used, when the system has more than 2 GB of real memory.

Pager is always in use. You must have confused it with another sub component of Virtual Memory Manager (VMM) -  Swap Manager.

Pager is a low level sub component of VMM, which is responsible for mapping (translation virtual address into phys address and similar op).
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Lars on December 13, 2020, 11:13:38 am
OS/4 user explained it very well.
Maybe I should have done that in the first place...
Yes, generic use of memory (without any application doing special things) above phys 4GB can be done in a 32-bit (logical address space) OS: if you completely rewrite OS/2's memory management as it exists today. And then to update all drivers and other kernel add-ons that insist on believing a physical address is limited to 32 bits.
About current performance of RAM DISK:
I'd have to look at the API again but I think it allows to copy data sectors to/from "normal" (user and system) memory spaces.
Anyone remember XMS (that kludge used by DOS) that allowed to copy to/from below 1 MB physical (real mode) address space from/to  high memory (reachable only in protected mode)?
It means an additional memcopy of each and every data sector you touch. This is necessary because PAE is not in general use in the existing memory management.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Lars on December 13, 2020, 11:22:50 am
Is it just me, but I find the ramdisk to be pretty slow, barely faster then my USB2 disk. Disk 4 is the Ramdisk and disk 5 is the USB3 on USB2 HD.
Code: [Select]
Hard disk 4: 255 sides, 1138 cylinders, 63 sectors per track = 8926 MB
Drive cache/bus transfer rate: 26649 k/sec
Data transfer rate on cylinder 0   : 23463 k/sec
Data transfer rate on cylinder 1136: 20314 k/sec
CPU usage by full speed disk transfers: 11%
Average data access time: Disk read error.
Multithreaded disk I/O (4 threads): 22193 k/sec, 12% CPU usage

Hard disk 5: 255 sides, 60085 cylinders, 255 sectors per track = 1907728 MB
Drive cache/bus transfer rate: 25500 k/sec
Data transfer rate on cylinder 0   : 21492 k/sec
Data transfer rate on cylinder 60083: 21287 k/sec
CPU usage by full speed disk transfers: 1%
Average data access time: 20.4 ms
Multithreaded disk I/O (4 threads): 21347 k/sec, 2% CPU usage

I had the impression that the PAE disk interface is actually implemented as a char device rather then a block device.
Architecturally,  an ADD driver qualifies as a char device because it has a driver name. But of course, it works with blocks of 512 byte sectors, just like any other ADD. ADDs introduced a special entry point into the driver to improve performance and simplify interface between DMDs and ADDs.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Andi B. on December 13, 2020, 11:48:46 am
Doug wrote -
Quote
but FAT32 is always slow, even in windows (and it has other limits,
Doug you wrote that numerous times but IMO you're wrong.

Copy paste of my measurements I posted somewhere else about USB3 speed -
Code: [Select]
Win7 to NTFS - 113 MByte/s
Win7 to FAT32 - similar

eCS, usb11.17 to JFS - 18 MByte/s
ArcaOS, usb12.07 to JFS - 17.5 MByte/s

( ArcaOS, usb12.07 to FAT32 - 0.13 Mbyte/s )
 
This figures are for writing a single large file to an external hard disk. These clearly show that for other OSes usually the speed of the external hard disk is the limiting factor. Not the filesystem (nor USB). At least with large files.

Only with OS/2 - ArcaOS we are limited by the drivers. And additionally, our FAT32 is exceptionally slow. These are problems of our drivers/OS and our FAT32 implementation. It does not help anyone when you repeat again and again that this may be a filesystem problem. So please stay on facts. Thanks.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Lars on December 13, 2020, 01:21:53 pm
Doug wrote -
Quote
but FAT32 is always slow, even in windows (and it has other limits,
Doug you wrote that numerous times but IMO you're wrong.

Copy paste of my measurements I posted somewhere else about USB3 speed -
Code: [Select]
Win7 to NTFS - 113 MByte/s
Win7 to FAT32 - similar

eCS, usb11.17 to JFS - 18 MByte/s
ArcaOS, usb12.07 to JFS - 17.5 MByte/s

( ArcaOS, usb12.07 to FAT32 - 0.13 Mbyte/s )
 
This figures are for writing a single large file to an external hard disk. These clearly show that for other OSes usually the speed of the external hard disk is the limiting factor. Not the filesystem (nor USB). At least with large files.

Only with OS/2 - ArcaOS we are limited by the drivers. And additionally, our FAT32 is exceptionally slow. These are problems of our drivers/OS and our FAT32 implementation. It does not help anyone when you repeat again and again that this may be a filesystem problem. So please stay on facts. Thanks.

This is remarkable, given the fact that one reason for the new line of 12.xx 32-bit device drivers was increase in performance. Surely does not look like it.
By the way: what measurement tool are you using ? I'd like to crosscheck with my own driver set.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Andi B. on December 13, 2020, 03:24:43 pm
Quote
By the way: what measurement tool are you using ? I'd like to crosscheck with my own driver set.
I usually take LarsenCommander on both OSes as it does the measurement automatically. LCMD also seems to wait until the cache is really written to the disk at the end of the copy process which is especially important on Win. Figures vary slightly when repeating tests a few times. But difference is usually no more than a few percent. To proof my results I also compared with standard copy command and stop watch. I measured from hitting enter until copy and eject finished. Taking into account the additional time eject took all my tests confirmed my initial numbers taken by simply copy from LarsenCommander are good enough to compare OSes/filesystems/drivers.

Cause I thought there is something wrong with my system when running ArcaOS I started the thread on the testers group. Unfortunately it seems ArcaOS and the drivers works as designed and there's no hope that it will get faster in future. At least not for my Intel based system. Maybe with other chip sets. But until now no one come up with better test results.

Btw. I made such tests in the past with your drivers too. They are a bit faster. But the difference is small. Using a different port (different internal chip) does make a lot of more difference. I remember my tests probably > 10 years ago where I could write to a external JFS hard disk with about 22MByte/s (USB2) while Win with the same hardware at that time did about 26MByte/s.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Dave Yeo on December 13, 2020, 06:05:46 pm
Here I used https://hobbes.nmsu.edu/download/pub/os2/util/benchmark/diskio118z.zip (https://hobbes.nmsu.edu/download/pub/os2/util/benchmark/diskio118z.zip) to benchmark my drives. It's old and could use a couple of updates, measurement of my fastest drives bus speed is negative so a variable needs to be bigger and it considers too fast of a seek time like the ram disk or SSD to be a disk error but it measures reading track 0 and the last track directly.
There is also a NT binary in the package.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: Dariusz Piatkowski on December 17, 2020, 03:44:43 pm
...It's old and could use a couple of updates, measurement of my fastest drives bus speed is negative so a variable needs to be bigger and it considers too fast of a seek time like the ram disk or SSD to be a disk error but it measures reading track 0 and the last track directly.
There is also a NT binary in the package.

Matching my experience, although the RAMDISK here fails to get most of the metrics:

Code: [Select]
Dhrystone 2.1 C benchmark routines (C) 1988 Reinhold P. Weicker
Dhrystone benchmark for this CPU: 2185936 runs/sec

Hard disk 1: 255 sides, 636 cylinders, 63 sectors per track = 4988 MB
Drive cache/bus transfer rate: 26754 k/sec
Data transfer rate on cylinder 0   : Disk read error.
Data transfer rate on cylinder 634 : Disk read error.
CPU usage by full speed disk transfers: Disk read error.
Average latency time: 0.0 ms
Average data access time: Disk read error.
Multithreaded disk I/O (4 threads): Disk read error.
-470888 k/sec, 29% CPU usage

...and numbers for something like my SSD show the obvious overflow errors:

Code: [Select]
Hard disk 2: 255 sides, 30401 cylinders, 63 sectors per track = 238472 MB
Drive cache/bus transfer rate: 115349 k/sec
Data transfer rate on cylinder 0   : -84103 k/sec
Data transfer rate on cylinder 30399: -133819 k/sec
CPU usage by full speed disk transfers: 24%
Average latency time: 0.1 ms
Average data access time: Disk read error.
Multithreaded disk I/O (4 threads): 115870 k/sec, 22% CPU usage

I did compile that source back in April of this year b/c I was trying to look at fixing those little things, and while that did give me a successfull build, more work needed to be devoted to that task...aha...Holidays are here...maybe now???  ;D
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: David Kiley on December 18, 2020, 01:17:24 pm
Since the OS/2 kernel is not open source, I always thought that some alternative for OS/2 kernel evolution issue will be to replace completely the OS/2 kernel for a different 64bit kernel and interpret the OS/2 personality with the IBM binaries over it. People will complain that the kernel is the OS, but I'm starting to believe that OS/2 is more about the experience of the desktop and being able to run the OS/2 applications software that we like and use.

Long term this probably will be the only choice if there is to be a 64bit OS/2. Being closed source there is always going to be roadblocks.
If someone took an open source kernel, recreating the WPS desktop experience, and then there could be some kind of "classic mode" like they did with OS X that would allow you to run legacy OS/2 apps. Moving to some open source base OS would also mean resources could be spent where they matter, instead of recreating the wheel such as with web browsers etc.

That being said ArcaOS and other developers have done a fantastic job keeping os/2 alive.
Title: Re: OS/2 - ArcaOS 64Bits Kernel Discussion
Post by: JTCaptKirk on December 18, 2020, 08:18:42 pm
Hi.  I seem to remember some scuttlebutt about there being an early work by IBM to  replace the OS/2 kernel with a version of the MACH kernel.  At the time they were working on something called the Workplace OS. It was also said by IBM that this kernel would have an "OS/2 Personality"  and perhaps the ability to have others too.  So with this is mind is there any
possibility that using the MACH kernel as a starting point could a 64 bit OS/2 be realized instead of starting from scratch  ??