Binary decompilation of OS2KRNL and other DLLs

[RickCHodgin]

A discussion cropped up on another site about a defunct software application (defunct since 2009) that has received new updates and bug fixes and a type of service pack with some new features / workarounds to existing limitations (written by someone who doesn't work for the company that abandoned it, but who is just a regular user out there with high technical skill), and the legality of U.S. users applying those patches.

It got me wondering if a technology, such as that related to OS/2 officially defunct now by IBM/Microsoft, could be legally decompiled from binary form for the purposes of education (learn how the kernel operates), for bug fixes / workarounds (vis-a-vis Arca Noae's path), or for other purposes?

Does the defunct state of a software app, even a major one, have any known ruling in a court case about what can be done with it since it is effectively abandonware by the original authoring / maintaining entity?

[Martin Iturbide]

Hi Rick

I think that the defunct state or being abandonware means nothing in legal terms.
What means something is the 107 clause on the US Copyright (Fair use).
- https://www.law.cornell.edu/uscode/text/17/107

I would understand that if it is for educational and non profit related work, it should fall under "fair use". And there is also a "effect of the use upon the potential market for or value of the copyrighted work", which I don't think will affect IBM sales.

But remember that on the "lawyer world" everyone can be sue, even if they don't have a case.

Regards

[RickCHodgin]

I think there may be a new avenue to arrive at the desired research, albeit completely indirectly.

A new ruling by the U.S. Copyright Office, which went into effect this morning at 12:00am (October 28, 2018), now makes it legal for security researchers to examine the contents of a software program to look for security flaws, and to issue repairs / patches which make the software more secure.

An entity could determine to publish a document outlining notable security flaws present in a piece of software, such as the OS2KRNL and related DLLs, by examining them for those purposes, as well as write a software program to patch / repair that software to remove the security flaws:  Section 1201 Acting Registers (see pages 71-84 for background discussion between proponents and opponents).

"Computer programs, in the form of firmware or software, regardless of the device on which they are run, when circumvention is initiated by the owner of the copy of the computer program or with the permission of the owner of the copy of the computer program, in order to conduct good-faith security research. This exemption does not obviate the need to comply with all other applicable laws and regulations."

Note:  It may be time for me to start a security research company.  Christopher Domas has been a long time party of interest for me in x86 research.  Maybe I could partner with his company.

In addition, if a library entity exists for the purpose of preserving historical software and/or hardware, they are now legally allowed to invade the software protection mechanisms and bypass original security or access limitations, but only for the purposes of preserving the work for their library.

This may allow EDM/2 to delve deeper into existing software for the purposes of historically documenting things related to OS/2, but only by entities operating under U.S. law.  I would be willing to volunteer for that purpose, sharing anything I find with EDM/2 for library / historical / legacy knowledge purposes.

[Martin Iturbide]

I would be willing to volunteer for that purpose, sharing anything I find with EDM/2 for library / historical / legacy knowledge purposes.

That would be excellent. Let me know when you want to add something on the EDM/2, I can help with that until Adrian gaves you access.

Regards

[RickCHodgin]

I would be willing to volunteer for that purpose, sharing anything I find with EDM/2 for library / historical / legacy knowledge purposes.

That would be excellent. Let me know when you want to add something on the EDM/2, I can help with that until Adrian gaves you access.

I'm going to start a 501c(3) corporation (non-profit entity) here in my home state, one dedicated solely to security research.  I will then begin working on identifying security flaws in OS/2 4.52 (specifically with SMP kernel versions) running on modern 64-bit CPUs (because people today will have multiple cores and will likely opt to run SMP versions).

I will publish my findings on my own website, as well as provide the information to EDM/2.

[RickCHodgin]

I briefly spoke with someone today and was informed that for the purposes of having a sole proprietorship, and for the explicit purposes of having a non-profit entity that is a sole proprietorship, I would not need to create a legal corporation, but am free to conduct business as me personally.

As such, I'll begin this work once I get the website registered for the security research division of Liberty Software Foundation.

My goals are a comprehensive analysis of IBM's 4.52 kernel and essential DLLs, looking specifically for weaknesses in the application loader, executable binary file formats, APIs (unused parameters, places where passing in purposefully crafted incorrect parameters might trigger unexpected behavior that could be exploited), weaknesses in the use of particular functions rather than other ones, potential buffer overrun errors, ways to introduce exploit code (which allows injection of binary code into the system to be run instead of the actual app's code), corruption of the stack, and other related issues, all through purposeful means.

I intend to create reports on each aspect of testing, and to provide the source code used to conduct all testing so my results can be verified by independent third parties.

[RickCHodgin]

I have found a complete 4.52 install online which is 4.52.14.086_W4.  I'll be using this as my baseline system.

If anyone has another suggestion, please let me know.  TYIA.

Note:  Here's the link to my LibSF Security Research website.

[Andy Willis]

I have found a complete 4.52 install online which is 4.52.14.086_W4.  I'll be using this as my baseline system.

If anyone has another suggestion, please let me know.  TYIA.

Note:  Here's the link to my LibSF Security Research website.

14.086_W4 is _not_ SMP capable (I mention because of your mentioning you wanted to work with the SMP.
The SMP kernel is so marked as - 14.201#@_SMP
14.201 is the current kernel in ArcaOS (cannot be sold separately to upgrade 4.52 or ecomstation). 
IBM's last generally available release was 14.104a (there was a 14.105 in a leaked testcase but it seemed to be a 14.104 build as it had the same bugs that 14.104a fixed).  14.106 was released by IBM only via special fixpak that required special (nearly impossible to obtain) service agreement.
This to say that 14.086 would be fine for learning but possibly of less practical value in terms of security as the same security holes may have been fixed.
Latest generally available kernels are at http://www.os2site.com/sw/upgrades/kernel/index.html
Your user agent has to show you are using OS/2 or you will have to request a userid and password (I had forgotten I had installed totalspoof and I was getting prompted for the password as a result).

[RickCHodgin]

14.086_W4 is _not_ SMP capable (I mention because of your mentioning you wanted to work with the SMP.
...
IBM's last generally available release was 14.104a (there was a 14.105 in a leaked testcase but it seemed to be a 14.104 build as it had the same bugs that 14.104a fixed).  14.106 was released by IBM only via special fixpak that required special (nearly impossible to obtain) service agreement.
This to say that 14.086 would be fine for learning but possibly of less practical value in terms of security as the same security holes may have been fixed.

Makes sense.

Latest generally available kernels are at http://www.os2site.com/sw/upgrades/kernel/index.html
Your user agent has to show you are using OS/2 or you will have to request a userid and password (I had forgotten I had installed totalspoof and I was getting prompted for the password as a result).

I've downloaded 14.104a and will use that one.  Thank you.

[Dave Yeo]

You can update the 4.52_w4 to SMP. One way is the package smp20050811.zip which should be where you found 104a. Basically the kernel (which can be changed from the included 105 to 104a_smp) the os2ldr and doscall1.dll. To actually use more then one cpu, you also need a PSD, usually ACPI.PSD from eCS or better Arca Noae, which also comes with some other files that have been patched to be SMP safe.

[RickCHodgin]

I've downloaded 14.104a and will use that one.  Thank you.

I can't get any of the SMP kernels to boot.  I am getting access violation errors (c0000005) in doscall1 right before it enters graphics mode.  In reading the comments on the SMP kernels, it seems the author was trying to fix those errors with the various revisions.

[Dave Yeo]

You are using the correct doscall1.dll?

[RickCHodgin]

You are using the correct doscall1.dll?

Should I install a particular fixpack then manually install the SMP kernel version I'm attempting over the top?

UPDATE:  I see XR_C006 mentioned for 4.52, with a 14.106 build level, but don't see a link.  I'm not sure what should be installed to get the latest version.  What would an internet user require to download 4.52 and the latest fix pack?

UPDATE2:  I was able to get email assistance, and see this archive has the fix packs.

[RickCHodgin]

The machine setup I'm going to attempt will be multi-boot with two DOS partitions, and one OS/2 partition.  A boot-time loader will allow me to choose which to boot into.

I will use the FAT32 OS/2 partition for the OS/2 install being researched.

One of the DOS partitions will be for all of the development tools I'll write for "debugging" OS/2.  These will include single-step boot debuggers in real-mode and protected mode, which are not part of OS/2 itself, but are my own tools (see here for protected mode and here for real mode), modified to capture information automatically as well as provide user-interaction-based debugging.

I will use the second DOS partition to capture and record runtime information.  It will also hold my binaries for custom booting into OS/2.

My first goal will be to create a type of Time Travel Debugging trace (see info on Microsoft's WinDbg and TTD), which allows me to capture everything that was done during boot.  This will allow me to perform a forensic analysis of everything that was performed during startup, to identify function entry points, oft-called intra-DLL targets, to identify which bits of code from which DLLs mapped into which areas of memory, etc.

Once that is complete, I'll design OS/2 tools which will allow me to create a custom loader, and intercept and redirect all DLL references, allowing me to perform the equivalent of a Windows procmon.exe capture for any apps of interest.

These are all goals.  We'll see what actually comes forth.  I'm looking forward to getting OS/2 to single-step boot.

[Andi B.]

...FAT32 OS/2 partition for the OS/2 install...

OS/2 can boot from FAT, HPFS and JFS. I've never heard it can boot from FAT32.