OS/2 - ArcaOS 64Bits Kernel Discussion

[Dave Yeo]

Is it just me, but I find the ramdisk to be pretty slow, barely faster then my USB2 disk. Disk 4 is the Ramdisk and disk 5 is the USB3 on USB2 HD.

Code: [Select]

Hard disk 4: 255 sides, 1138 cylinders, 63 sectors per track = 8926 MB
Drive cache/bus transfer rate: 26649 k/sec
Data transfer rate on cylinder 0   : 23463 k/sec
Data transfer rate on cylinder 1136: 20314 k/sec
CPU usage by full speed disk transfers: 11%
Average data access time: Disk read error.
Multithreaded disk I/O (4 threads): 22193 k/sec, 12% CPU usage

Hard disk 5: 255 sides, 60085 cylinders, 255 sectors per track = 1907728 MB
Drive cache/bus transfer rate: 25500 k/sec
Data transfer rate on cylinder 0   : 21492 k/sec
Data transfer rate on cylinder 60083: 21287 k/sec
CPU usage by full speed disk transfers: 1%
Average data access time: 20.4 ms
Multithreaded disk I/O (4 threads): 21347 k/sec, 2% CPU usage

I had the impression that the PAE disk interface is actually implemented as a char device rather then a block device.

[OS4User]

Just  to clarify few things.

Almost all user and system code operates with virtual memory only.

Processor in a 32 bit system can map physical memory  into the virtual memory (paging)  in 3 ways:
- "32-bit"  32 bit phys address  (4Gb max address)
- PAE       64 bit phys address
- PSE36   36 bit phys address  (64Gb max address)

OS/2 kernel  uses "32-bit" paging thus can address 4Gb of phys mem only.  To use other paging mode it is nessasery to rewrite at least component Pager in the kernel  and almost all drivers (some of them expect that  phys address  can be 32bit only).

To rewrite Pager is a really huge job.  I'm quite skeptical that it is possible on enthusiasm only.

[Doug Bissett]

Doug, just try to understand what Lars wrote.

What Lars wrote is probably correct, but that doesn't mean that it can't be changed, if somebody tries. Before QSINIT was created, most people were saying "it can't be done". It was done. now get over it,and try to improve it.

Is it just me, but I find the ramdisk to be pretty slow, barely faster then my USB2 disk. Disk 4 is the Ramdisk and disk 5 is the USB3 on USB2 HD.

I find that it depends on what file system you use. FAT16 is far faster than any of the other options (but has other limits, that make it not suitable). FAT32, is the slowest, but FAT32 is always slow, even in windows (and it has other limits, that make it not suitable). I usually use HPFS, and try to turn off as much of the cache, and lazy write, options as I can. It would probably be much better if the cache could be totally eliminated (why cache something that is going to memory anyway? It is just unnecessary overhead). I have used JFS on the RAMDISK, but then it must use the JFS cache, which actually slows it down, and you need the cache for other disks, so you can't just turn it off.

To rewrite Pager is a really huge job.  I'm quite skeptical that it is possible on enthusiasm only.

Who said anything about "Pager"? Pager is hardly ever used, when the system has more than 2 GB of real memory. That can also be put on the RAMDISK (using HPFS), but if you have 3.5 GB of real memory, available to OS/2, Pager will never be used anyway. What we need, is to offload the shared memory stuff, somehow. Private memory could also use it, if there is an access method (perhaps that would be easier).

[OS4User]

To rewrite Pager is a really huge job.  I'm quite skeptical that it is possible on enthusiasm only.

Who said anything about "Pager"? Pager is hardly ever used, when the system has more than 2 GB of real memory.

Pager is always in use. You must have confused it with another sub component of Virtual Memory Manager (VMM) -  Swap Manager.

Pager is a low level sub component of VMM, which is responsible for mapping (translation virtual address into phys address and similar op).

[Lars]

OS/4 user explained it very well.
Maybe I should have done that in the first place...
Yes, generic use of memory (without any application doing special things) above phys 4GB can be done in a 32-bit (logical address space) OS: if you completely rewrite OS/2's memory management as it exists today. And then to update all drivers and other kernel add-ons that insist on believing a physical address is limited to 32 bits.
About current performance of RAM DISK:
I'd have to look at the API again but I think it allows to copy data sectors to/from "normal" (user and system) memory spaces.
Anyone remember XMS (that kludge used by DOS) that allowed to copy to/from below 1 MB physical (real mode) address space from/to  high memory (reachable only in protected mode)?
It means an additional memcopy of each and every data sector you touch. This is necessary because PAE is not in general use in the existing memory management.

[Lars]

Is it just me, but I find the ramdisk to be pretty slow, barely faster then my USB2 disk. Disk 4 is the Ramdisk and disk 5 is the USB3 on USB2 HD.

Code: [Select]

Hard disk 4: 255 sides, 1138 cylinders, 63 sectors per track = 8926 MB
Drive cache/bus transfer rate: 26649 k/sec
Data transfer rate on cylinder 0   : 23463 k/sec
Data transfer rate on cylinder 1136: 20314 k/sec
CPU usage by full speed disk transfers: 11%
Average data access time: Disk read error.
Multithreaded disk I/O (4 threads): 22193 k/sec, 12% CPU usage

Hard disk 5: 255 sides, 60085 cylinders, 255 sectors per track = 1907728 MB
Drive cache/bus transfer rate: 25500 k/sec
Data transfer rate on cylinder 0   : 21492 k/sec
Data transfer rate on cylinder 60083: 21287 k/sec
CPU usage by full speed disk transfers: 1%
Average data access time: 20.4 ms
Multithreaded disk I/O (4 threads): 21347 k/sec, 2% CPU usage

I had the impression that the PAE disk interface is actually implemented as a char device rather then a block device.

Architecturally,  an ADD driver qualifies as a char device because it has a driver name. But of course, it works with blocks of 512 byte sectors, just like any other ADD. ADDs introduced a special entry point into the driver to improve performance and simplify interface between DMDs and ADDs.

[Andi B.]

Doug wrote -

but FAT32 is always slow, even in windows (and it has other limits,

Doug you wrote that numerous times but IMO you're wrong.

Copy paste of my measurements I posted somewhere else about USB3 speed -

Code: [Select]

Win7 to NTFS - 113 MByte/s
Win7 to FAT32 - similar

eCS, usb11.17 to JFS - 18 MByte/s
ArcaOS, usb12.07 to JFS - 17.5 MByte/s

( ArcaOS, usb12.07 to FAT32 - 0.13 Mbyte/s )
 This figures are for writing a single large file to an external hard disk. These clearly show that for other OSes usually the speed of the external hard disk is the limiting factor. Not the filesystem (nor USB). At least with large files.

Only with OS/2 - ArcaOS we are limited by the drivers. And additionally, our FAT32 is exceptionally slow. These are problems of our drivers/OS and our FAT32 implementation. It does not help anyone when you repeat again and again that this may be a filesystem problem. So please stay on facts. Thanks.

[Lars]

Doug wrote -

but FAT32 is always slow, even in windows (and it has other limits,

Doug you wrote that numerous times but IMO you're wrong.

Copy paste of my measurements I posted somewhere else about USB3 speed -

Code: [Select]

Win7 to NTFS - 113 MByte/s
Win7 to FAT32 - similar

eCS, usb11.17 to JFS - 18 MByte/s
ArcaOS, usb12.07 to JFS - 17.5 MByte/s

( ArcaOS, usb12.07 to FAT32 - 0.13 Mbyte/s )
 This figures are for writing a single large file to an external hard disk. These clearly show that for other OSes usually the speed of the external hard disk is the limiting factor. Not the filesystem (nor USB). At least with large files.

Only with OS/2 - ArcaOS we are limited by the drivers. And additionally, our FAT32 is exceptionally slow. These are problems of our drivers/OS and our FAT32 implementation. It does not help anyone when you repeat again and again that this may be a filesystem problem. So please stay on facts. Thanks.

This is remarkable, given the fact that one reason for the new line of 12.xx 32-bit device drivers was increase in performance. Surely does not look like it.
By the way: what measurement tool are you using ? I'd like to crosscheck with my own driver set.

[Andi B.]

By the way: what measurement tool are you using ? I'd like to crosscheck with my own driver set.

I usually take LarsenCommander on both OSes as it does the measurement automatically. LCMD also seems to wait until the cache is really written to the disk at the end of the copy process which is especially important on Win. Figures vary slightly when repeating tests a few times. But difference is usually no more than a few percent. To proof my results I also compared with standard copy command and stop watch. I measured from hitting enter until copy and eject finished. Taking into account the additional time eject took all my tests confirmed my initial numbers taken by simply copy from LarsenCommander are good enough to compare OSes/filesystems/drivers.

Cause I thought there is something wrong with my system when running ArcaOS I started the thread on the testers group. Unfortunately it seems ArcaOS and the drivers works as designed and there's no hope that it will get faster in future. At least not for my Intel based system. Maybe with other chip sets. But until now no one come up with better test results.

Btw. I made such tests in the past with your drivers too. They are a bit faster. But the difference is small. Using a different port (different internal chip) does make a lot of more difference. I remember my tests probably > 10 years ago where I could write to a external JFS hard disk with about 22MByte/s (USB2) while Win with the same hardware at that time did about 26MByte/s.

[Dave Yeo]

Here I used https://hobbes.nmsu.edu/download/pub/os2/util/benchmark/diskio118z.zip to benchmark my drives. It's old and could use a couple of updates, measurement of my fastest drives bus speed is negative so a variable needs to be bigger and it considers too fast of a seek time like the ram disk or SSD to be a disk error but it measures reading track 0 and the last track directly.
There is also a NT binary in the package.

[Dariusz Piatkowski]

...It's old and could use a couple of updates, measurement of my fastest drives bus speed is negative so a variable needs to be bigger and it considers too fast of a seek time like the ram disk or SSD to be a disk error but it measures reading track 0 and the last track directly.
There is also a NT binary in the package.

Matching my experience, although the RAMDISK here fails to get most of the metrics:

Code: [Select]

Dhrystone 2.1 C benchmark routines (C) 1988 Reinhold P. Weicker
Dhrystone benchmark for this CPU: 2185936 runs/sec

Hard disk 1: 255 sides, 636 cylinders, 63 sectors per track = 4988 MB
Drive cache/bus transfer rate: 26754 k/sec
Data transfer rate on cylinder 0   : Disk read error.
Data transfer rate on cylinder 634 : Disk read error.
CPU usage by full speed disk transfers: Disk read error.
Average latency time: 0.0 ms
Average data access time: Disk read error.
Multithreaded disk I/O (4 threads): Disk read error.
-470888 k/sec, 29% CPU usage

...and numbers for something like my SSD show the obvious overflow errors:

Code: [Select]

Hard disk 2: 255 sides, 30401 cylinders, 63 sectors per track = 238472 MB
Drive cache/bus transfer rate: 115349 k/sec
Data transfer rate on cylinder 0   : -84103 k/sec
Data transfer rate on cylinder 30399: -133819 k/sec
CPU usage by full speed disk transfers: 24%
Average latency time: 0.1 ms
Average data access time: Disk read error.
Multithreaded disk I/O (4 threads): 115870 k/sec, 22% CPU usage

I did compile that source back in April of this year b/c I was trying to look at fixing those little things, and while that did give me a successfull build, more work needed to be devoted to that task...aha...Holidays are here...maybe now??? 

[David Kiley]

Since the OS/2 kernel is not open source, I always thought that some alternative for OS/2 kernel evolution issue will be to replace completely the OS/2 kernel for a different 64bit kernel and interpret the OS/2 personality with the IBM binaries over it. People will complain that the kernel is the OS, but I'm starting to believe that OS/2 is more about the experience of the desktop and being able to run the OS/2 applications software that we like and use.

Long term this probably will be the only choice if there is to be a 64bit OS/2. Being closed source there is always going to be roadblocks.
If someone took an open source kernel, recreating the WPS desktop experience, and then there could be some kind of "classic mode" like they did with OS X that would allow you to run legacy OS/2 apps. Moving to some open source base OS would also mean resources could be spent where they matter, instead of recreating the wheel such as with web browsers etc.

That being said ArcaOS and other developers have done a fantastic job keeping os/2 alive.

[JTCaptKirk]

Hi.  I seem to remember some scuttlebutt about there being an early work by IBM to  replace the OS/2 kernel with a version of the MACH kernel.  At the time they were working on something called the Workplace OS. It was also said by IBM that this kernel would have an "OS/2 Personality"  and perhaps the ability to have others too.  So with this is mind is there any
possibility that using the MACH kernel as a starting point could a 64 bit OS/2 be realized instead of starting from scratch  ??