BENCHMARKING Windows and OS/2: Difference between revisions

From OS2World.Com Wiki
Jump to navigation Jump to search
Created page with "<PRE> BENCHMARKING Windows and OS/2 December 11, 1992 by Timothy F. Sipples [This message is available as file winbench.txt, available via anonymous ftp from ftp-os2.nmsu.edu..."
 
Jugbogdan (talk | contribs)
mNo edit summary
Line 1: Line 1:
<PRE>
''December 11, 1992 - by [[Timothy F. Sipples]]''
BENCHMARKING Windows and OS/2
December 11, 1992
by Timothy F. Sipples


[This message is available as file winbench.txt, available via
[This message is available as file winbench.txt, available via anonymous ftp from ftp-os2.nmsu.edu on the Internet. Followups may be
anonymous ftp from ftp-os2.nmsu.edu on the Internet. Followups may be
directed to comp.os.os2.misc, but please redirect subsequent followups to comp.os.os2.advocacy as content warrants.]
directed to comp.os.os2.misc, but please redirect subsequent followups
to comp.os.os2.advocacy as content warrants.]


The following is a summary of results of the PC Magazine Winbench
The following is a summary of results of the PC Magazine Winbench Version 2.51 benchmarking software when run under Windows 3.1 on
Version 2.51 benchmarking software when run under Windows 3.1 on
DOS 5.02, Windows 3.1 NT (October public beta, CD-ROM), and OS/2 2.0x (November Professional Developer's Kit CD-ROM, the third public Win-OS/2 3.1 beta).
DOS 5.02, Windows 3.1 NT (October public beta, CD-ROM), and OS/2 2.0x
(November Professional Developer's Kit CD-ROM, the third public Win-OS/2
3.1 beta).


Note that Microsoft does not recommend using NT currently for performance
Note that Microsoft does not recommend using NT currently for performance benchmarking. Also please note that these are a first set of
benchmarking. Also please note that these are a first set of
benchmark figures for one particular benchmark. I hope to have numbers on a mixed sample Excel macro and on a disk I/O test.  However, these numbers may be instructive.
benchmark figures for one particular benchmark. I hope to have numbers
on a mixed sample Excel macro and on a disk I/O test.  However, these
numbers may be instructive.


Insofar as possible an effort was made to compare apples to apples.
Insofar as possible an effort was made to compare apples to apples. Tests were conducted by IBM under the supervision of another customer
Tests were conducted by IBM under the supervision of another customer
and myself.
and myself.


The hardware consisted of an IBM PS/2 Model 95 with an 80486DX 33 MHz
The hardware consisted of an IBM PS/2 Model 95 with an 80486DX 33 MHz processor, 16 MB of RAM, XGA(-1) graphics adapter, and a 320 MB, 12 ms SCSI hard drive. (Aside from the addition of a CD-ROM drive and Sound Blaster adapter, this machine was a stock Model 95, including
processor, 16 MB of RAM, XGA(-1) graphics adapter, and a 320 MB, 12 ms
a busmastering 32-bit SCSI cache adapter. A standard PS/2 mouse was attached and was not moved while the benchmark was running.)  (Other
SCSI hard drive. (Aside from the addition of a CD-ROM drive and
tests were performed on an IBM PS/2 Model 57 with standard VGA for comparison. These results are outlined in brief, below.)
SoundBlaster adapter, this machine was a stock Model 95, including
a busmastering 32-bit SCSI cache adapter. A standard PS/2 mouse was
attached and was not moved while the benchmark was running.)  (Other
tests were performed on an IBM PS/2 Model 57 with standard VGA for
comparison. These results are outlined in brief, below.)


Note that Winbench measures graphics performance using the default
Note that Winbench measures graphics performance using the default weighting system from PC Magazine (and not set by IBM). As such, the
weighting system from PC Magazine (and not set by IBM). As such, the
hard disk details are not particularly relevant, since disk I/O throughput is not being measured. (With luck, I will try to obtain
hard disk details are not particularly relevant, since disk I/O
those results and release them in the near future.)  Winbench runs entirely from memory, so disk I/O is not a factor. This benchmark
throughput is not being measured. (With luck, I will try to obtain
measures pure graphics performance. However, on the slightest chance that disk I/O would skew any results, Winbench was loaded from the
those results and release them in the near future.)  Winbench runs
same location on disk for performance measurement under each operating system (environment). Also, just for the record, the hard disk was
entirely from memory, so disk I/O is not a factor. This benchmark
partitioned with 200 MB to OS/2 2.0x and 100 MB to Windows NT and Windows 3.1 on DOS 5.02. All partitions were FAT. To install Windows NT
measures pure graphics performance. However, on the slightest chance
Beta and OS/2 2.0x on the same hard disk required zeroing out the ID of the OS/2 Boot Manager partition. Windows NT Beta includes a specific check which is deliberately designed to refuse installation if OS/2 Boot Manager is present on the hard disk. (OS/2 2.0x includes no such NT check.)  Zeroing out the Boot Manager ID renders NT Beta installable while preserving OS/2 Boot Manager capabilities.
that disk I/O would skew any results, Winbench was loaded from the
same location on disk for performance measurement under each operating
system (environment). Also, just for the record, the hard disk was
partitioned with 200 MB to OS/2 2.0x and 100 MB to Windows NT and
Windows 3.1 on DOS 5.02. All partitions were FAT. To install Windows NT
Beta and OS/2 2.0x on the same hard disk required zeroing out the ID
of the OS/2 Boot Manager partition. Windows NT Beta includes a specific
check which is deliberately designed to refuse installation if OS/2
Boot Manager is present on the hard disk. (OS/2 2.0x includes no such
NT check.)  Zeroing out the Boot Manager ID renders NT Beta installable
while preserving OS/2 Boot Manager capabilities.


All system defaults at install time were used. (Exception: OS/2 2.0x
All system defaults at install time were used. (Exception: OS/2 2.0x had PRIORITY_DISK_IO set to NO, FILES=100, and PRINTMONBUF=402,0,0.
had PRIORITY_DISK_IO set to NO, FILES=100, and PRINTMONBUF=402,0,0.
These settings are disk and printing related and do not impact Winbench. Also, DOS was loaded HIGH for the OS/2 2.0x DOS sessions, but every other default was retained for the Win-OS/2 sessions. The DOS HIGH setting also should not impact Winbench. DOS HIGH was the default on the other two.) Winbench was the only task running on each system when run.
These settings are disk and printing related and do not impact Winbench.
Also, DOS was loaded HIGH for the OS/2 2.0x DOS sessions, but every other
default was retained for the Win-OS/2 sessions. The DOS HIGH setting
also should not impact Winbench. DOS HIGH was the default on the other
two.) Winbench was the only task running on each system when run.


Windows 3.1 on DOS 5.02 was installed with SMARTDRV using all default
Windows 3.1 on DOS 5.02 was installed with SMARTDRV using all default install time settings. Windows NT was installed with all default settings. All used 1024x768 in 256 colors for their desktops.
install time settings. Windows NT was installed with all default settings.
All used 1024x768 in 256 colors for their desktops.


[The "Optimized" OS/2 2.0 figure consists of the following changes to the
[The "Optimized" OS/2 2.0 figure consists of the following changes to the session's DOS Settings: VIDEO_RETRACE_EMULATION from ON to OFF, EMS zeroed/disabled, XMS_MEMORY_LIMIT to 64K, IDLE_SENSITIVITY to 100, DDE and Clipboard changed from Public to Private, and VIDEO_8514A_XGA_IOTRAP from ON to OFF. These changes were made more to satisfy my curiosity than anything else.]
session's DOS Settings: VIDEO_RETRACE_EMULATION from ON to OFF, EMS zeroed/
disabled, XMS_MEMORY_LIMIT to 64K, IDLE_SENSITIVITY to 100, DDE and
Clipboard changed from Public to Private, and VIDEO_8514A_XGA_IOTRAP from
ON to OFF. These changes were made more to satisfy my curiosity than
anything else.]


Winbench supplies a final index number called the Graphics WinMark. This
Winbench supplies a final index number called the Graphics WinMark. This figure is the one presented below.
figure is the one presented below.


As a baseline comparison, Windows 3.1 on DOS 5.0 on a Compaq 386/25e
As a baseline comparison, Windows 3.1 on DOS 5.0 on a Compaq 386/25e running with standard VGA at 640x480 in 16 colors generates 1,676,475
running with standard VGA at 640x480 in 16 colors generates 1,676,475
WinMarks. (The higher the number of WinMarks, the faster the graphics performance.)
WinMarks. (The higher the number of WinMarks, the faster the graphics
performance.)


Double trials were run in certain cases when the results seemed
Double trials were run in certain cases when the results seemed surprising. These double trials also give a sense of the small
surprising. These double trials also give a sense of the small
variance inherent in the Winbench benchmark.
variance inherent in the Winbench benchmark.


Here are the results:
Here are the results:
 
{|class="wikitable"
Windows 3.1 on DOS 5.02, 386 Enhanced Mode 4,154,629
|Windows 3.1 on DOS 5.02, 386 Enhanced Mode||align="right"|4,154,629
Windows 3.1 on DOS 5.02, Standard Mode 4,149,752
|-
OS/2 2.0x, Full Screen Win-OS/2 5,523,993
|Windows 3.1 on DOS 5.02, Standard Mode||align="right"|4,149,752
OS/2 2.0x, FS Win-OS/2, "Optimized" (two trials) 5,600,009
|-
5,585,557
|OS/2 2.0x, Full Screen Win-OS/2||align="right"|5,523,993
OS/2 2.0x, "Seamless" Win-OS/2 NOT TESTED
|-
(Winbench does not operate under the current Win-OS/2 beta in
|OS/2 2.0x, FS Win-OS/2, "Optimized" (two trials)||align="right"|5,600,009<br/>5,585,557
"seamless" mode.)
|-
Windows NT (two trials) 1,146,638
|OS/2 2.0x, "Seamless" Win-OS/2||NOT TESTED
1,147,051
|-
 
|colspan=2|(Winbench does not operate under the current Win-OS/2 beta in "seamless" mode.)
Exact numbers are not available to me yet, but on a stock IBM PS/2
|-
Model 57 with 16 MB of RAM and standard VGA, the relative Winmarks
|Windows NT (two trials)||align="right"|1,146,638<br/>1,147,051
|}
Exact numbers are not available to me yet, but on a stock IBM PS/2 Model 57 with 16 MB of RAM and standard VGA, the relative Winmarks
were approximately as follows:
were approximately as follows:
{|
|Windows 3.1 on DOS 5.02||100%
|-
|OS/2 2.0x, FS Win-OS/2, Not "Optimized"||95%
|-
|Windows NT Beta||40%
|}


Windows 3.1 on DOS 5.02 100%
==Preliminary Conclusions==
OS/2 2.0x, FS Win-OS/2, Not "Optimized" 95%
The major surprise to me was the sluggish graphics performance in Windows NT Beta, which is borne out by subjective perceptions as well.
Windows NT Beta 40%
(I was heard to exclaim, "This is much faster" when the switch was made from Windows NT Beta to Windows 3.1 on DOS 5.02.)
 
 
PRELIMINARY CONCLUSIONS


The major surprise to me was the sluggish graphics performance in
At first I suspected that the peculiarity was confined to the XGA driver (perhaps a particularly bad implementation). But the VGA numbers
Windows NT Beta, which is borne out by subjective perceptions as well.
seem to confirm that it isn't a driver issue (although it does point out that OS/2 takes good advantage of busmastered coprocessed video).
(I was heard to exclaim, "This is much faster" when the switch was
(As another reference point, Windows 3.1 on DOS 5.02 generated 3,187,612 Winmarks on the same IBM Model 95 when it was dropped down to VGA
made from Windows NT Beta to Windows 3.1 on DOS 5.02.)
640x480 16 color mode.  Thus, Windows 3.1 on DOS benefits from XGA as well.) The Model 57 (not the 486SLC model, which was not tested) uses
relatively slow, vanilla VGA hardware, and yet NT Beta suffers dramatically while OS/2 holds its own with Windows 3.1 for DOS, so it seems that OS/2's good showing is not due to any peculiarity with XGA.


At first I suspected that the peculiarity was confined to the XGA
In any event, the clearest conclusion one can draw from these numbers, it seems, is that Windows NT Beta has a long way to go toward even
driver (perhaps a particularly bad implementation).  But the VGA numbers
coming close to the graphics performance in either OS/2 2.0x Win-OS/2 or Windows 3.1 on DOS.
seem to confirm that it isn't a driver issue (although it does point
out that OS/2 takes good advantage of busmastered coprocessed video).
(As another reference point, Windows 3.1 on DOS 5.02 generated 3,187,612
Winmarks on the same IBM Model 95 when it was dropped down to VGA
640x480 16 color mode. Thus, Windows 3.1 on DOS benefits from XGA as
well.)  The Model 57 (not the 486SLC model, which was not tested) uses
relatively slow, vanilla VGA hardware, and yet NT Beta suffers dramatically
while OS/2 holds its own with Windows 3.1 for DOS, so it seems that
OS/2's good showing is not due to any peculiarity with XGA.


In any event, the clearest conclusion one can draw from these numbers,
One might argue that these numbers are based on a 16-bit Windows benchmark, and that NT might really shine with a 32-bit benchmark.
it seems, is that Windows NT Beta has a long way to go toward even
Perhaps true, but when benchmarking one must try and relate the benchmarks back to reality.  The vast installed base of PC software
coming close to the graphics performance in either OS/2 2.0x Win-OS/2
is 16-bit, and as users migrate to a new 32-bit operating system they will carry with them lots of 16-bit software.  It will be some time,
or Windows 3.1 on DOS.
even under the rosiest scenarios, before any large portion of that software base is replaced with the 32-bit versions.  And that 32-bit
version could just as easily (if not more easily, since OS/2 2.0 has been a released, shrinkwrapped product for many months with hundreds
of shipping 32-bit applications) be an OS/2 2.0 32-bit application as it could be a Windows NT 32-bit application.


One might argue that these numbers are based on a 16-bit Windows
One might also argue that running Win-OS/2 full screen is overstating the OS/2 2.0x performance in relation to NT, since NT is doing something similar to OS/2 (in running Windows 16-bit applications "seamless") and since users will often run Windows applications "seamless" under OS/2 2.0x.
benchmark, and that NT might really shine with a 32-bit benchmark.
Perhaps, and while the seamless Win-OS/2 numbers are not presented above, at least OS/2 2.0x gives the option of providing a mode which executes existing 16-bit Windows applications as fast as possible (in a desktop which is just like Windows for DOS - or Windows NT, for that matter - namely with a full screen Windows desktop) as well as the lower performance, "seamless" mode. NT does not offer both modes of operation - there is but one desktop.
Perhaps true, but when benchmarking one must try and relate the
benchmarks back to reality.  The vast installed base of PC software
is 16-bit, and as users migrate to a new 32-bit operating system they
will carry with them lots of 16-bit software. It will be some time,
even under the rosiest scenarios, before any large portion of that
software base is replaced with the 32-bit versions.  And that 32-bit
version could just as easily (if not more easily, since OS/2 2.0 has
been a released, shrinkwrapped product for many months with hundreds
of shipping 32-bit applications) be an OS/2 2.0 32-bit application as
it could be a Windows NT 32-bit application.


One might also argue that running Win-OS/2 full screen is overstating
Then there is the issue of standard v. enhanced mode, and OS/2's capabilities in running applications which require WINMEM32.DLL. The benchmark numbers were provided for Windows 3.1 in Standard Mode on DOS 5.02, for comparison, so that apples can be compared to apples.  (In certain areas, Windows 3.1 Standard Mode is faster than 386 Enhanced Mode, but this is not reflected in this particular benchmark.)  The services provided by WINMEM32.DLL will not be available under Windows NT - such applications will not run in the future under either operating system. This consideration is important when comparing the ease with which one can migrate to a 32-bit operating system.
the OS/2 2.0x performance in relation to NT, since NT is doing something
(The version of OS/2 2.0x under consideration can run with Windows real mode and Windows 3.0 kernels, as an option, simultaneously with the Windows 3.1 kernel, to provide better backward compatibility than Windows 3.1.)
similar to OS/2 (in running Windows 16-bit applications "seamless") and
since users will often run Windows applications "seamless" under OS/2 2.0x.
Perhaps, and while the seamless Win-OS/2 numbers are not presented above,
at least OS/2 2.0x gives the option of providing a mode which executes
existing 16-bit Windows applications as fast as possible (in a desktop which
is just like Windows for DOS -- or Windows NT, for that matter -- namely
with a full screen Windows desktop) as well as the lower performance,
"seamless" mode. NT does not offer both modes of operation -- there is
but one desktop.


Then there is the issue of standard v. enhanced mode, and OS/2's capabilities
These results are preliminary and based on beta code. And the Winbench measure has its flaws. However, without dramatic performance improvements in Windows NT, OS/2 2.0 seems to offer dramatically better performance for the large library of 16-bit Windows 2.x and 3.x applications, thus protecting that substantial software investmentWith XGA, OS/2 2.0x even outpaces Windows 3.1 for DOS in raw graphics performance, as measured in Winmarks. Windows NT Beta, on the other hand, when running on a speedy PS/2 Model 95, lags well behind even the baseline Compaq 386/25e in standard VGA, which is a system that is roughly half the speed of the Model 95 in almost every dimension.
in running applications which require WINMEM32.DLL.  The benchmark numbers
were provided for Windows 3.1 in Standard Mode on DOS 5.02, for comparison,
so that apples can be compared to apples(In certain areas, Windows 3.1
Standard Mode is faster than 386 Enhanced Mode, but this is not reflected in
this particular benchmark.)  The services provided by WINMEM32.DLL
will not be available under Windows NT -- such applications will not run in
the future under either operating system.  This consideration is important when
comparing the ease with which one can migrate to a 32-bit operating system.
(The version of OS/2 2.0x under consideration can run with Windows real
mode and Windows 3.0 kernels, as an option, simultaneously with the Windows
3.1 kernel, to provide better backward compatibility than Windows 3.1.)


These results are preliminary and based on beta code.  And the Winbench
Note also that these tests were conducted on a 16 MB system, which, if anything, skews the results in favor of Windows NT Beta. The results may well have been even more striking with an 8 MB system, a more typical memory configuration. Windows NT Beta suffers performance problems on "low" RAM configurations (and, in this case, 8 MB qualifies). OS/2 2.0x, on the other hand, is very comfortable with such a RAM configuration. Winbench, when running on such a "constrained" system, may well have caused NT to page, thus impacting the results.
measure has its flaws.  However, without dramatic performance improvements
in Windows NT, OS/2 2.0 seems to offer dramatically better performance for
the large library of 16-bit Windows 2.x and 3.x applications, thus protecting
that substantial software investment. With XGA, OS/2 2.0x even outpaces
Windows 3.1 for DOS in raw graphics performance, as measured in Winmarks.
Windows NT Beta, on the other hand, when running on a speedy PS/2 Model 95,
lags well behind even the baseline Compaq 386/25e in standard VGA, which is
a system that is roughly half the speed of the Model 95 in almost every
dimension.


Note also that these tests were conducted on a 16 MB system, which, if anything,
I hope my description of the methodology used was clear, and I hope to be providing additional numbers in the near future.
skews the results in favor of Windows NT Beta.  The results may
well have been even more striking with an 8 MB system, a more typical memory
configuration.  Windows NT Beta suffers performance problems on "low" RAM
configurations (and, in this case, 8 MB qualifies).  OS/2 2.0x, on the other
hand, is very comfortable with such a RAM configuration.  Winbench, when
running on such a "constrained" system, may well have caused NT to page, thus
impacting the results.
 
I hope my description of the methodology used was clear, and I hope to be
providing additional numbers in the near future.


T.F.S.
T.F.S.


 
[[Category:OS/2 2.0]]
</PRE>

Revision as of 16:38, 26 November 2018

December 11, 1992 - by Timothy F. Sipples

[This message is available as file winbench.txt, available via anonymous ftp from ftp-os2.nmsu.edu on the Internet. Followups may be directed to comp.os.os2.misc, but please redirect subsequent followups to comp.os.os2.advocacy as content warrants.]

The following is a summary of results of the PC Magazine Winbench Version 2.51 benchmarking software when run under Windows 3.1 on DOS 5.02, Windows 3.1 NT (October public beta, CD-ROM), and OS/2 2.0x (November Professional Developer's Kit CD-ROM, the third public Win-OS/2 3.1 beta).

Note that Microsoft does not recommend using NT currently for performance benchmarking. Also please note that these are a first set of benchmark figures for one particular benchmark. I hope to have numbers on a mixed sample Excel macro and on a disk I/O test. However, these numbers may be instructive.

Insofar as possible an effort was made to compare apples to apples. Tests were conducted by IBM under the supervision of another customer and myself.

The hardware consisted of an IBM PS/2 Model 95 with an 80486DX 33 MHz processor, 16 MB of RAM, XGA(-1) graphics adapter, and a 320 MB, 12 ms SCSI hard drive. (Aside from the addition of a CD-ROM drive and Sound Blaster adapter, this machine was a stock Model 95, including a busmastering 32-bit SCSI cache adapter. A standard PS/2 mouse was attached and was not moved while the benchmark was running.) (Other tests were performed on an IBM PS/2 Model 57 with standard VGA for comparison. These results are outlined in brief, below.)

Note that Winbench measures graphics performance using the default weighting system from PC Magazine (and not set by IBM). As such, the hard disk details are not particularly relevant, since disk I/O throughput is not being measured. (With luck, I will try to obtain those results and release them in the near future.) Winbench runs entirely from memory, so disk I/O is not a factor. This benchmark measures pure graphics performance. However, on the slightest chance that disk I/O would skew any results, Winbench was loaded from the same location on disk for performance measurement under each operating system (environment). Also, just for the record, the hard disk was partitioned with 200 MB to OS/2 2.0x and 100 MB to Windows NT and Windows 3.1 on DOS 5.02. All partitions were FAT. To install Windows NT Beta and OS/2 2.0x on the same hard disk required zeroing out the ID of the OS/2 Boot Manager partition. Windows NT Beta includes a specific check which is deliberately designed to refuse installation if OS/2 Boot Manager is present on the hard disk. (OS/2 2.0x includes no such NT check.) Zeroing out the Boot Manager ID renders NT Beta installable while preserving OS/2 Boot Manager capabilities.

All system defaults at install time were used. (Exception: OS/2 2.0x had PRIORITY_DISK_IO set to NO, FILES=100, and PRINTMONBUF=402,0,0. These settings are disk and printing related and do not impact Winbench. Also, DOS was loaded HIGH for the OS/2 2.0x DOS sessions, but every other default was retained for the Win-OS/2 sessions. The DOS HIGH setting also should not impact Winbench. DOS HIGH was the default on the other two.) Winbench was the only task running on each system when run.

Windows 3.1 on DOS 5.02 was installed with SMARTDRV using all default install time settings. Windows NT was installed with all default settings. All used 1024x768 in 256 colors for their desktops.

[The "Optimized" OS/2 2.0 figure consists of the following changes to the session's DOS Settings: VIDEO_RETRACE_EMULATION from ON to OFF, EMS zeroed/disabled, XMS_MEMORY_LIMIT to 64K, IDLE_SENSITIVITY to 100, DDE and Clipboard changed from Public to Private, and VIDEO_8514A_XGA_IOTRAP from ON to OFF. These changes were made more to satisfy my curiosity than anything else.]

Winbench supplies a final index number called the Graphics WinMark. This figure is the one presented below.

As a baseline comparison, Windows 3.1 on DOS 5.0 on a Compaq 386/25e running with standard VGA at 640x480 in 16 colors generates 1,676,475 WinMarks. (The higher the number of WinMarks, the faster the graphics performance.)

Double trials were run in certain cases when the results seemed surprising. These double trials also give a sense of the small variance inherent in the Winbench benchmark.

Here are the results:

Windows 3.1 on DOS 5.02, 386 Enhanced Mode 4,154,629
Windows 3.1 on DOS 5.02, Standard Mode 4,149,752
OS/2 2.0x, Full Screen Win-OS/2 5,523,993
OS/2 2.0x, FS Win-OS/2, "Optimized" (two trials) 5,600,009
5,585,557
OS/2 2.0x, "Seamless" Win-OS/2 NOT TESTED
(Winbench does not operate under the current Win-OS/2 beta in "seamless" mode.)
Windows NT (two trials) 1,146,638
1,147,051

Exact numbers are not available to me yet, but on a stock IBM PS/2 Model 57 with 16 MB of RAM and standard VGA, the relative Winmarks were approximately as follows:

Windows 3.1 on DOS 5.02 100%
OS/2 2.0x, FS Win-OS/2, Not "Optimized" 95%
Windows NT Beta 40%

Preliminary Conclusions

The major surprise to me was the sluggish graphics performance in Windows NT Beta, which is borne out by subjective perceptions as well. (I was heard to exclaim, "This is much faster" when the switch was made from Windows NT Beta to Windows 3.1 on DOS 5.02.)

At first I suspected that the peculiarity was confined to the XGA driver (perhaps a particularly bad implementation). But the VGA numbers seem to confirm that it isn't a driver issue (although it does point out that OS/2 takes good advantage of busmastered coprocessed video). (As another reference point, Windows 3.1 on DOS 5.02 generated 3,187,612 Winmarks on the same IBM Model 95 when it was dropped down to VGA 640x480 16 color mode. Thus, Windows 3.1 on DOS benefits from XGA as well.) The Model 57 (not the 486SLC model, which was not tested) uses relatively slow, vanilla VGA hardware, and yet NT Beta suffers dramatically while OS/2 holds its own with Windows 3.1 for DOS, so it seems that OS/2's good showing is not due to any peculiarity with XGA.

In any event, the clearest conclusion one can draw from these numbers, it seems, is that Windows NT Beta has a long way to go toward even coming close to the graphics performance in either OS/2 2.0x Win-OS/2 or Windows 3.1 on DOS.

One might argue that these numbers are based on a 16-bit Windows benchmark, and that NT might really shine with a 32-bit benchmark. Perhaps true, but when benchmarking one must try and relate the benchmarks back to reality. The vast installed base of PC software is 16-bit, and as users migrate to a new 32-bit operating system they will carry with them lots of 16-bit software. It will be some time, even under the rosiest scenarios, before any large portion of that software base is replaced with the 32-bit versions. And that 32-bit version could just as easily (if not more easily, since OS/2 2.0 has been a released, shrinkwrapped product for many months with hundreds of shipping 32-bit applications) be an OS/2 2.0 32-bit application as it could be a Windows NT 32-bit application.

One might also argue that running Win-OS/2 full screen is overstating the OS/2 2.0x performance in relation to NT, since NT is doing something similar to OS/2 (in running Windows 16-bit applications "seamless") and since users will often run Windows applications "seamless" under OS/2 2.0x. Perhaps, and while the seamless Win-OS/2 numbers are not presented above, at least OS/2 2.0x gives the option of providing a mode which executes existing 16-bit Windows applications as fast as possible (in a desktop which is just like Windows for DOS - or Windows NT, for that matter - namely with a full screen Windows desktop) as well as the lower performance, "seamless" mode. NT does not offer both modes of operation - there is but one desktop.

Then there is the issue of standard v. enhanced mode, and OS/2's capabilities in running applications which require WINMEM32.DLL. The benchmark numbers were provided for Windows 3.1 in Standard Mode on DOS 5.02, for comparison, so that apples can be compared to apples. (In certain areas, Windows 3.1 Standard Mode is faster than 386 Enhanced Mode, but this is not reflected in this particular benchmark.) The services provided by WINMEM32.DLL will not be available under Windows NT - such applications will not run in the future under either operating system. This consideration is important when comparing the ease with which one can migrate to a 32-bit operating system. (The version of OS/2 2.0x under consideration can run with Windows real mode and Windows 3.0 kernels, as an option, simultaneously with the Windows 3.1 kernel, to provide better backward compatibility than Windows 3.1.)

These results are preliminary and based on beta code. And the Winbench measure has its flaws. However, without dramatic performance improvements in Windows NT, OS/2 2.0 seems to offer dramatically better performance for the large library of 16-bit Windows 2.x and 3.x applications, thus protecting that substantial software investment. With XGA, OS/2 2.0x even outpaces Windows 3.1 for DOS in raw graphics performance, as measured in Winmarks. Windows NT Beta, on the other hand, when running on a speedy PS/2 Model 95, lags well behind even the baseline Compaq 386/25e in standard VGA, which is a system that is roughly half the speed of the Model 95 in almost every dimension.

Note also that these tests were conducted on a 16 MB system, which, if anything, skews the results in favor of Windows NT Beta. The results may well have been even more striking with an 8 MB system, a more typical memory configuration. Windows NT Beta suffers performance problems on "low" RAM configurations (and, in this case, 8 MB qualifies). OS/2 2.0x, on the other hand, is very comfortable with such a RAM configuration. Winbench, when running on such a "constrained" system, may well have caused NT to page, thus impacting the results.

I hope my description of the methodology used was clear, and I hope to be providing additional numbers in the near future.

T.F.S.