OS2 World Community Forum

Public Discussions => General Discussion => Topic started by: RickCHodgin on October 29, 2018, 01:38:18 pm

Title: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 29, 2018, 01:38:18 pm
A discussion cropped up on another site about a defunct software application (defunct since 2009) that has received new updates and bug fixes and a type of service pack with some new features / workarounds to existing limitations (written by someone who doesn't work for the company that abandoned it, but who is just a regular user out there with high technical skill), and the legality of U.S. users applying those patches.

It got me wondering if a technology, such as that related to OS/2 officially defunct now by IBM/Microsoft, could be legally decompiled from binary form for the purposes of education (learn how the kernel operates), for bug fixes / workarounds (vis-a-vis Arca Noae's path), or for other purposes?

Does the defunct state of a software app, even a major one, have any known ruling in a court case about what can be done with it since it is effectively abandonware by the original authoring / maintaining entity?
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Martin Iturbide on October 29, 2018, 03:02:50 pm
Hi Rick

I think that the defunct state or being abandonware means nothing in legal terms.
What means something is the 107 clause on the US Copyright (Fair use).
- https://www.law.cornell.edu/uscode/text/17/107 (https://www.law.cornell.edu/uscode/text/17/107)

I would understand that if it is for educational and non profit related work, it should fall under "fair use". And there is also a "effect of the use upon the potential market for or value of the copyrighted work", which I don't think will affect IBM sales.

But remember that on the "lawyer world" everyone can be sue, even if they don't have a case.

Regards
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 29, 2018, 03:26:38 pm
I think there may be a new avenue to arrive at the desired research, albeit completely indirectly.

A new ruling by the U.S. Copyright Office, which went into effect this morning at 12:00am (October 28, 2018), now makes it legal for security researchers to examine the contents of a software program to look for security flaws, and to issue repairs / patches which make the software more secure.

An entity could determine to publish a document outlining notable security flaws present in a piece of software, such as the OS2KRNL and related DLLs, by examining them for those purposes, as well as write a software program to patch / repair that software to remove the security flaws:  Section 1201 Acting Registers (https://www.copyright.gov/1201/2018/2018_Section_1201_Acting_Registers_Recommendation.pdf) (see pages 71-84 for background discussion between proponents and opponents).

"Computer programs, in the form of firmware or software, regardless of the device on which they are run, when circumvention is initiated by the owner of the copy of the computer program or with the permission of the owner of the copy of the computer program, in order to conduct good-faith security research. This exemption does not obviate the need to comply with all other applicable laws and regulations."

Note:  It may be time for me to start a security research company.  Christopher Domas has been a long time party of interest for me in x86 research.  Maybe I could partner with his company.

In addition, if a library entity exists for the purpose of preserving historical software and/or hardware, they are now legally allowed to invade the software protection mechanisms and bypass original security or access limitations, but only for the purposes of preserving the work for their library.

This may allow EDM/2 to delve deeper into existing software for the purposes of historically documenting things related to OS/2, but only by entities operating under U.S. law.  I would be willing to volunteer for that purpose, sharing anything I find with EDM/2 for library / historical / legacy knowledge purposes.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Martin Iturbide on October 29, 2018, 03:51:53 pm
I would be willing to volunteer for that purpose, sharing anything I find with EDM/2 for library / historical / legacy knowledge purposes.

That would be excellent. Let me know when you want to add something on the EDM/2, I can help with that until Adrian gaves you access.

Regards
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 29, 2018, 04:04:23 pm
I would be willing to volunteer for that purpose, sharing anything I find with EDM/2 for library / historical / legacy knowledge purposes.

That would be excellent. Let me know when you want to add something on the EDM/2, I can help with that until Adrian gaves you access.

I'm going to start a 501c(3) corporation (non-profit entity) here in my home state, one dedicated solely to security research.  I will then begin working on identifying security flaws in OS/2 4.52 (specifically with SMP kernel versions) running on modern 64-bit CPUs (because people today will have multiple cores and will likely opt to run SMP versions).

I will publish my findings on my own website, as well as provide the information to EDM/2.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 29, 2018, 08:31:09 pm
I briefly spoke with someone today and was informed that for the purposes of having a sole proprietorship, and for the explicit purposes of having a non-profit entity that is a sole proprietorship, I would not need to create a legal corporation, but am free to conduct business as me personally.

As such, I'll begin this work once I get the website registered for the security research division of Liberty Software Foundation.

My goals are a comprehensive analysis of IBM's 4.52 kernel and essential DLLs, looking specifically for weaknesses in the application loader, executable binary file formats, APIs (unused parameters, places where passing in purposefully crafted incorrect parameters might trigger unexpected behavior that could be exploited), weaknesses in the use of particular functions rather than other ones, potential buffer overrun errors, ways to introduce exploit code (which allows injection of binary code into the system to be run instead of the actual app's code), corruption of the stack, and other related issues, all through purposeful means.

I intend to create reports on each aspect of testing, and to provide the source code used to conduct all testing so my results can be verified by independent third parties.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 01:06:54 am
I have found a complete 4.52 install online which is 4.52.14.086_W4.  I'll be using this as my baseline system.

If anyone has another suggestion, please let me know.  TYIA.

Note:  Here's the link to my LibSF Security Research website (http://www.libsf-security-research.org).
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Andy Willis on October 31, 2018, 02:07:09 am
I have found a complete 4.52 install online which is 4.52.14.086_W4.  I'll be using this as my baseline system.

If anyone has another suggestion, please let me know.  TYIA.

Note:  Here's the link to my LibSF Security Research website (http://www.libsf-security-research.org).
14.086_W4 is _not_ SMP capable (I mention because of your mentioning you wanted to work with the SMP.
The SMP kernel is so marked as - 14.201#@_SMP
14.201 is the current kernel in ArcaOS (cannot be sold separately to upgrade 4.52 or ecomstation). 
IBM's last generally available release was 14.104a (there was a 14.105 in a leaked testcase but it seemed to be a 14.104 build as it had the same bugs that 14.104a fixed).  14.106 was released by IBM only via special fixpak that required special (nearly impossible to obtain) service agreement.
This to say that 14.086 would be fine for learning but possibly of less practical value in terms of security as the same security holes may have been fixed.
Latest generally available kernels are at http://www.os2site.com/sw/upgrades/kernel/index.html
Your user agent has to show you are using OS/2 or you will have to request a userid and password (I had forgotten I had installed totalspoof and I was getting prompted for the password as a result).
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 02:41:25 am
14.086_W4 is _not_ SMP capable (I mention because of your mentioning you wanted to work with the SMP.
...
IBM's last generally available release was 14.104a (there was a 14.105 in a leaked testcase but it seemed to be a 14.104 build as it had the same bugs that 14.104a fixed).  14.106 was released by IBM only via special fixpak that required special (nearly impossible to obtain) service agreement.
This to say that 14.086 would be fine for learning but possibly of less practical value in terms of security as the same security holes may have been fixed.

Makes sense.

Quote
Latest generally available kernels are at http://www.os2site.com/sw/upgrades/kernel/index.html
Your user agent has to show you are using OS/2 or you will have to request a userid and password (I had forgotten I had installed totalspoof and I was getting prompted for the password as a result).

I've downloaded 14.104a and will use that one.  Thank you.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Dave Yeo on October 31, 2018, 03:49:25 am
You can update the 4.52_w4 to SMP. One way is the package smp20050811.zip which should be where you found 104a. Basically the kernel (which can be changed from the included 105 to 104a_smp) the os2ldr and doscall1.dll. To actually use more then one cpu, you also need a PSD, usually ACPI.PSD from eCS or better Arca Noae, which also comes with some other files that have been patched to be SMP safe.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 03:57:47 am
I've downloaded 14.104a and will use that one.  Thank you.

I can't get any of the SMP kernels to boot.  I am getting access violation errors (c0000005) in doscall1 right before it enters graphics mode.  In reading the comments on the SMP kernels, it seems the author was trying to fix those errors with the various revisions.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Dave Yeo on October 31, 2018, 05:02:14 am
You are using the correct doscall1.dll?
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 10:46:06 am
You are using the correct doscall1.dll?

Should I install a particular fixpack then manually install the SMP kernel version I'm attempting over the top?

UPDATE:  I see XR_C006 mentioned for 4.52, with a 14.106 build level, but don't see a link.  I'm not sure what should be installed to get the latest version.  What would an internet user require to download 4.52 and the latest fix pack?

UPDATE2:  I was able to get email assistance, and see this archive (https://archive.org/download/OS2Warp4.52) has the fix packs.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 01:12:06 pm
The machine setup I'm going to attempt will be multi-boot with two DOS partitions, and one OS/2 partition.  A boot-time loader will allow me to choose which to boot into.

I will use the FAT32 OS/2 partition for the OS/2 install being researched.

One of the DOS partitions will be for all of the development tools I'll write for "debugging" OS/2.  These will include single-step boot debuggers in real-mode and protected mode, which are not part of OS/2 itself, but are my own tools (see here for protected mode (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/exodus/source/debi) and here for real mode (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/exodus/source/x86)), modified to capture information automatically as well as provide user-interaction-based debugging.

I will use the second DOS partition to capture and record runtime information.  It will also hold my binaries for custom booting into OS/2.

My first goal will be to create a type of Time Travel Debugging trace (see info on Microsoft's WinDbg and TTD (https://www.youtube.com/watch?v=BVslyei0804)), which allows me to capture everything that was done during boot.  This will allow me to perform a forensic analysis of everything that was performed during startup, to identify function entry points, oft-called intra-DLL targets, to identify which bits of code from which DLLs mapped into which areas of memory, etc.

Once that is complete, I'll design OS/2 tools which will allow me to create a custom loader, and intercept and redirect all DLL references, allowing me to perform the equivalent of a Windows procmon.exe capture for any apps of interest.

These are all goals.  We'll see what actually comes forth.  I'm looking forward to getting OS/2 to single-step boot.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Andi B. on October 31, 2018, 01:47:04 pm
Quote
...FAT32 OS/2 partition for the OS/2 install...
OS/2 can boot from FAT, HPFS and JFS. I've never heard it can boot from FAT32.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Sigurd Fastenrath on October 31, 2018, 02:23:30 pm
Once that is complete, I'll design OS/2 tools which will allow me to create a custom loader, and intercept and redirect all DLL references, allowing me to perform the equivalent of a Windows procmon.exe capture for any apps of interest.

To be honest: I do not understand why you try to reinvent the Wheel. I would like to Point you to the already existing, I would say THE OS/2 LOADER, where ArcaOS seems to base on as well, QSINIT. I see no reason to create another one. And, as others already stated, I would like to suggest that you take part in already existing OS/2 Projects, to concentrate on one Problem instead of writing everything from Scratch. Even for IBM with (I do not know exactly the number) some hundred programmers I would guess in Hand, it took years to get something out the door. I would guess that some Projects/other developers would be happy to Profit from your Knowledge and abbilities. This also may Benefit your Health and Prevent from falling back in some Kind of burn out Syndrom.
All the best.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 02:30:57 pm
Quote
...FAT32 OS/2 partition for the OS/2 install...
OS/2 can boot from FAT, HPFS and JFS. I've never heard it can boot from FAT32.

Quite correct.  My mistake.  It's just FAT.  During installation you see the OS/2 format option.  Under file Windows disk management you see the disk partition info.  I had just assumed its "FAT" reference was FAT32.  Kind of freaky / surprising that it's not.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 02:34:29 pm
Once that is complete, I'll design OS/2 tools which will allow me to create a custom loader, and intercept and redirect all DLL references, allowing me to perform the equivalent of a Windows procmon.exe capture for any apps of interest.

To be honest: I do not understand why you try to reinvent the Wheel. I would like to Point you to the already existing, I would say THE OS/2 LOADER, where ArcaOS seems to base on as well, QSINIT. I see no reason to create another one. And, as others already stated, I would like to suggest that you take part in already existing OS/2 Projects, to concentrate on one Problem instead of writing everything from Scratch. Even for IBM with (I do not know exactly the number) some hundred programmers I would guess in Hand, it took years to get something out the door. I would guess that some Projects/other developers would be happy to Profit from your Knowledge and abbilities. This also may Benefit your Health and Prevent from falling back in some Kind of burn out Syndrom.
All the best.

Nearly everyone in my life tells me this.  But deep and permeating within my heart is a goal to create something which expressly and purposefully honors the flag and banner I pursue with my life.  It's not done for the project, or money, or fame.  It's done because I have these low-level development abilities, and an understanding of x86 hardware and OS design, and I wish to contribute something toward those ends.  But, I don't want to just do it to get it done.  I want to do it for that purpose I'm in pursuit of.

If you'd like to move to another forum I'll explain everything, but the bottom line is I'm not just a doer ... I'm a purposeful, conscientious doer, and I'm doing it for a real reason that is exceedingly strong within me.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Dave Yeo on October 31, 2018, 04:55:26 pm
FAT is probably a bad choice for an install volume due to its limits. 8.3 names (the WPS supports long names in extended attributes but the kernel can only see 8.3) along with the 2GB partition limit, large clusters and such. HPFS might be a good choice for you, there are DOS HPFS drivers if you need to access the OS/2 partition from DOS.
You also need an OS/2 aware boot manager if booting from anywhere but C: as OS/2 needs to know where it is installed. You can use the supplied OS/2 Bootmanager, it needs a primary partition of 1 cylinder.
As for fixpaks, they're installed after you get the system running. And doscall1.dll is matched to the kernel somewhat, you need a SMP doscall1.dll to go with the SMP kernel, which internally is a DLL and includes doscall0.

OS/2 also includes a lot of debugging stuff, including tools to trace stuff as you're describing. There's also sym files that you could disassemble to get the correct symbol names of various functions.
Good luck, it's a huge job you're undertaking.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 05:10:28 pm
FAT is probably a bad choice for an install volume due to its limits. 8.3 names (the WPS supports long names in extended attributes but the kernel can only see 8.3) along with the 2GB partition limit, large clusters and such. HPFS might be a good choice for you, there are DOS HPFS drivers if you need to access the OS/2 partition from DOS.
You also need an OS/2 aware boot manager if booting from anywhere but C: as OS/2 needs to know where it is installed. You can use the supplied OS/2 Bootmanager, it needs a primary partition of 1 cylinder.
As for fixpaks, they're installed after you get the system running. And doscall1.dll is matched to the kernel somewhat, you need a SMP doscall1.dll to go with the SMP kernel, which internally is a DLL and includes doscall0.

OS/2 also includes a lot of debugging stuff, including tools to trace stuff as you're describing. There's also sym files that you could disassemble to get the correct symbol names of various functions.
Good luck, it's a huge job you're undertaking.

I appreciate any help you offer, Dave.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Dave Yeo on October 31, 2018, 05:54:21 pm
I'd suggest installing OS/2 a couple of times in different ways, eg different partitions, different file systems and such just to get a feel for it. Installing in a FAT partition involves a different boot procedure then using an IFS (installable file system), installing on an extended partition means having a bootloader to start the boot process and pass on to the kernel where it is booting from so it can load its drivers. And so on.
You might also want to look at airboot https://trac.netlabs.org/air-boot (https://trac.netlabs.org/air-boot) a GPLv3 bootloader written in assembly and aimed at OS/2 which lives in the MBR area.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 06:03:55 pm
I'd suggest installing OS/2 a couple of times in different ways, eg different partitions, different file systems and such just to get a feel for it. Installing in a FAT partition involves a different boot procedure then using an IFS (installable file system), installing on an extended partition means having a bootloader to start the boot process and pass on to the kernel where it is booting from so it can load its drivers. And so on.
You might also want to look at airboot https://trac.netlabs.org/air-boot (https://trac.netlabs.org/air-boot) a GPLv3 bootloader written in assembly and aimed at OS/2 which lives in the MBR area.

I have installed it multiple times.  I have primarily used HPFS.  The reason I chose FAT this time was because I needed the ability to easily access the partition from other OSes (primarily DOS, Windows, and Linux).  I've never installed the HPFS driver for DOS.  Since I can do that I'll switch.

I'll work on installing the boot loader.  I used it back in the 90s with Warp 4 on my machine (DOS + OS/2 partitions).

For my own kernel, I developed code which allowed me to debug the boot sector code by putting the debugger code at known sequential locations on the disk, and then baking those into the boot loader so it just iterated to load Nn sectors continuously, and then did some interrupt descriptor table updates to handle INT1 and INT3.  I then issued INT3 to allow me to single-step through.  I only used bare bones BIOS functions, and it worked nicely.  That's the real mode debugger (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/exodus/source/x86) I mentioned previously.  I used this pb.ini (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/exodus/source/pb.ini) (put boot) file to determine what to write, and wrote it to disk with this utility (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/exodus/source/pb).  Not my best work, but it worked.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Dave Yeo on October 31, 2018, 06:20:35 pm
Various HPFS drivers for DOS here, http://hobbes.nmsu.edu/h-browse.php?dir=/pub/dos (http://hobbes.nmsu.edu/h-browse.php?dir=/pub/dos). Linux includes a HPFS read/write module though not all dists install it by default. I believe most Debian based dists do. NT used to be able to access HPFS through the pinball driver, capability probably went away after XP. Don't use it on partitions over 4.3GBs.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 06:54:14 pm
Various HPFS drivers for DOS here, http://hobbes.nmsu.edu/h-browse.php?dir=/pub/dos (http://hobbes.nmsu.edu/h-browse.php?dir=/pub/dos). Linux includes a HPFS read/write module though not all dists install it by default. I believe most Debian based dists do. NT used to be able to access HPFS through the pinball driver, capability probably went away after XP. Don't use it on partitions over 4.3GBs.

My real goal is the tracing.  If you can help me with any kernel debuggers, setting up the ability to record any traces, or other such features, that would be great.  I had intended to use the sym files to obtain meaningful names for the kernel version I will use.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Dave Yeo on October 31, 2018, 07:06:51 pm
Well there's the icat kernel debugger, http://hobbes.nmsu.edu/download/pub/os2/dev/util/icos0602.zip (http://hobbes.nmsu.edu/download/pub/os2/dev/util/icos0602.zip). I believe ideally you need 2 boxes with a serial connection to use it. Perhaps start here, http://home.earthlink.net/~steve53/os2diags/index.html (http://home.earthlink.net/~steve53/os2diags/index.html) for more info on using it.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 07:11:22 pm
Well there's the icat kernel debugger, http://hobbes.nmsu.edu/download/pub/os2/dev/util/icos0602.zip (http://hobbes.nmsu.edu/download/pub/os2/dev/util/icos0602.zip). I believe ideally you need 2 boxes with a serial connection to use it. Perhaps start here, http://home.earthlink.net/~steve53/os2diags/index.html (http://home.earthlink.net/~steve53/os2diags/index.html) for more info on using it.

Has anyone ever set it up with two VMs?
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Dave Yeo on October 31, 2018, 09:26:08 pm
Don't know. Don't see why it wouldn't work.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Lars on October 31, 2018, 09:49:36 pm
Well there's the icat kernel debugger, http://hobbes.nmsu.edu/download/pub/os2/dev/util/icos0602.zip (http://hobbes.nmsu.edu/download/pub/os2/dev/util/icos0602.zip). I believe ideally you need 2 boxes with a serial connection to use it. Perhaps start here, http://home.earthlink.net/~steve53/os2diags/index.html (http://home.earthlink.net/~steve53/os2diags/index.html) for more info on using it.

Has anyone ever set it up with two VMs?
I am Using it from a VM but the target is a real OS/2 Machine. I am Using it With UDP but Then you cannot debug System Startup.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Valery Sedletski on October 31, 2018, 10:48:50 pm
2AndiB: OS/2 can boot from FAT32 (and from FAT/VFAT/exFAT too) with my FreeLDR bootloader. (and sysinstx command for fat32.ifs installs FreeLDR boot block on a medium). If someone interested, there is Team BOOT/2 boot disk, in which creation I participated. It is able to boot from FAT/FAT32/exFAT/HPFS/JFS/CDFS/etc. Can be installed on a LiveCD, as well as on a boot flash stick/USB harddisk/ATA harddisk/etc. It supports VirtualPC and VBox additions, so it is good for testing OS/2 in VM's. The latest version is here: ftp://osfree.org/upload/ramfsos2-20161021.zip

PS: 2Rick Hodgin: last OS/2 port of VBox I did supports redirecting the guest COM port to a named pipe. You can connect to this named pipe with DTerm terminal emulator (which supports listening the named pipes). This way, you can work with guest OS/2 kernel kernel debugger. So, you could debug the OS/2 kernel in VBox, and avoid using two real machines for debugging (a debugee machine and a terminal machine connected to its COM port with a null-modem cable)

PPS: DTerm can be downloaded from its author's page at http://os2acw.googlepages.com/. (Unfortunately, the author has been died, and no more new versions, and no sources. But it is a very good terminal emulator for kernel debugging).

PPPS: If you have VBox, but on Linux or Windows host, you can use miniterm terminal emulator on Linux, and Putty on Windows. You can choose "Host pipe" option here too. In Linux case, a local socket is used really, instead of a pipe. In Windows case, it uses a named pipe too. OS/2 pipes are specified as \pipe\dterm in both DTerm settings and VBox settings. Windows pipes are specified as \\.\pipe\somepipe. In Linux, sockets are created in /tmp. So, you specify /tmp/vboxfifo in VBox, and "unix#/tmp/vboxfifo" in minicom. In all three cases, it's working quite well.

PPPPS: And BTW, FAT is supported by fat32.ifs, including VFAT long file names. So, 8.3 limits are no more applicable. So, OS/2 sees the long file names on FAT without problems.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on October 31, 2018, 11:55:49 pm
PS: 2Rick Hodgin: last OS/2 port of VBox I did supports redirecting the guest COM port to a named pipe. You can connect to this named pipe with DTerm terminal emulator (which supports listening the named pipes). This way, you can work with guest OS/2 kernel kernel debugger. So, you could debug the OS/2 kernel in VBox, and avoid using two real machines for debugging (a debugee machine and a terminal machine connected to its COM port with a null-modem cable)

PPS: DTerm can be downloaded from its author's page at http://os2acw.googlepages.com/. (Unfortunately, the author has been died, and no more new versions, and no sources. But it is a very good terminal emulator for kernel debugging).

PPPS: If you have VBox, but on Linux or Windows host, you can use miniterm terminal emulator on Linux, and Putty on Windows. You can choose "Host pipe" option here too. In Linux case, a local socket is used really, instead of a pipe. In Windows case, it uses a named pipe too. OS/2 pipes are specified as \pipe\dterm in both DTerm settings and VBox settings. Windows pipes are specified as \\.\pipe\somepipe. In Linux, sockets are created in /tmp. So, you specify /tmp/vboxfifo in VBox, and "unix#/tmp/vboxfifo" in minicom. In all three cases, it's working quite well.

I will need help getting that setup.  What I'm seeking to do is use external tools to examine and map the internal workings of the kernel and related system drivers / DLLs, and for the express purpose of finding out where it is subject to weaknesses or flaws resulting in potential data loss.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Valery Sedletski on November 01, 2018, 12:02:56 am
PPPPS: And BTW, FAT is supported by fat32.ifs, including VFAT long file names. So, 8.3 limits are no more applicable. So, OS/2 sees the long file names on FAT without problems.

And, there are HPFS drivers for Linux and Windows. The HPFS driver was removed from WinXP/win2k3, but it can be added there back. Look here: http://hobbes.nmsu.edu/download/pub/os2/util/disk/hpfs_w2k.zip, it is a HPFS driver from win2k, which can be installed in WinXP. There are limitations of 4 GB for max HPFS volume size. (You should look into the readme for instructions)

For debugging bootsectors and loaders, I'd also suggest you to not reinvent wheels, but use existing debuggers of existing virtual machines. There are built-in debuggers in Bochs and VirtualBox. For writing my own boot sectors, I used the first of the two.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on November 01, 2018, 01:14:17 pm
I couldn't get HPFS to be recognized in Windows.  What driver do I need to install?

I tried the latest one I could find, which was an NT 4 driver.  I tried it in Windows Server 2003, and it would only recognize the drive as existing, but saw no partitions on it, no file system.  It reported 0 bytes on disk, 0 bytes available.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Dave Yeo on November 01, 2018, 03:33:36 pm
The NT4 driver (pinball.sys) is the latest. I only ever used it in W2K. I know it has a 4.3 GB limit or so for partitions, don't know if it has limitations for disk size.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on November 01, 2018, 04:14:28 pm
The NT4 driver (pinball.sys) is the latest. I only ever used it in W2K. I know it has a 4.3 GB limit or so for partitions, don't know if it has limitations for disk size.

For the work I've been doing, I've been limiting my partitions to 2GB.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Valery Sedletski on November 01, 2018, 05:37:57 pm
2Rick C. Hodgin: You're probably trying to install the archive called hpfs_nt4.zip, but you need an updated package, http://hobbes.nmsu.edu/download/pub/os2/util/disk/hpfs_w2k.zip. MS removed a required API from NT kernel after win2k, so some workaround is required on newer kernels, so they require an updated package. (Though, I did not tried to install it myself, so I forgot exact details (they should be in the readme). I only heard of such problems from this package creator. It is _dixie_, who is also QSINIT author. All credits to him.).
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on November 01, 2018, 05:48:36 pm
2Rick C. Hodgin: You're probably trying to install the archive called hpfs_nt4.zip, but you need an updated package, http://hobbes.nmsu.edu/download/pub/os2/util/disk/hpfs_w2k.zip. MS removed a required API from NT kernel after win2k, so some workaround is required on newer kernels, so they require an updated package. (Though, I did not tried to install it myself, so I forgot exact details (they should be in the readme). I only heard of such problems from this package creator. It is _dixie_, who is also QSINIT author. All credits to him.).

In my family we have an expression we say to one another when we're helpful.  It's a play on the German word "danke" for thank you, but we say "donkey." :-)

Donkey, Valery.  Will check it out tonight.  Donkey. :-)
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on November 01, 2018, 09:07:56 pm
For debugging bootsectors and loaders, I'd also suggest you to not reinvent wheels, but use existing debuggers of existing virtual machines. There are built-in debuggers in Bochs and VirtualBox. For writing my own boot sectors, I used the first of the two.

I've tried using Bochs' built-in debugger before, but I've never been able to get it to work properly.  So what I've done is create my own debugger interface in Bochs using the 0xb0000 memory block for monochrome graphics.  It uses a named pipe to transfer data to/from Bochs and my eternal "MoMo" (MOnochrome MOnitor) app.

See this video:  http://www.visual-freepro.org/videos/2014_02_13__exodus_debi_debugger.ogv

And here's the source code:  http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/exodus/bochs/source/2.3.5/MoMo
(Clone the repo, open that .sln file, and you'll see it all there)

And here:  http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/exodus/bochs/source/2.3.5/bochs-2.3.5/momo

It is a true protected-mode graphical debugger running in the code in the VM.  I have a similar debugger that works in text mode for the boot sector, and displays similarly, but much more simply.

If you're on IRC and can help me with Bochs' debugger I'd appreciate it.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: RickCHodgin on November 01, 2018, 11:30:43 pm
http://hobbes.nmsu.edu/download/pub/os2/util/disk/hpfs_w2k.zip

It worked.  I had to get PINBALL.SYS from \windows\system32\drivers\ and copy it to my unzip folder, then run BUPDATE PINBALL.BKG and it gave me a new PINBALL.SYS, which I copied with RS.DLL back to the \windows\system32\drivers\ folder.  Then, after reboot my OS/2 partitions were visible.

Thank you again, Valery.
Title: Re: Binary decompilation of OS2KRNL and other DLLs
Post by: Valery Sedletski on November 03, 2018, 02:20:25 am
Good, need to try pinball.sys on WinXP myself too :)

The bochs version I'm using is compiled by Froloff some years ago. But since then, he no
more uses OS/2 and his site is down. Though, I uploaded it to my ftp:

Quote
ftp://osfree.org/upload/froloff/bochs-froloff.zip

You just configure .bochsrc, add the disk images and configure the
boot device order, from which device to load first (boot: disk to boot
from a harddisk). Then you start bochs.cmd in a cmd/4os2 window
and it drops into the debugger.

Bochs is fine for debugging boot sectors or loaders, but it is too slow
to run a complex OS like OS/2 is. Not sure if it will start at all completely.