Binary decompilation of OS2KRNL and other DLLs

[Sigurd Fastenrath]

Once that is complete, I'll design OS/2 tools which will allow me to create a custom loader, and intercept and redirect all DLL references, allowing me to perform the equivalent of a Windows procmon.exe capture for any apps of interest.

To be honest: I do not understand why you try to reinvent the Wheel. I would like to Point you to the already existing, I would say THE OS/2 LOADER, where ArcaOS seems to base on as well, QSINIT. I see no reason to create another one. And, as others already stated, I would like to suggest that you take part in already existing OS/2 Projects, to concentrate on one Problem instead of writing everything from Scratch. Even for IBM with (I do not know exactly the number) some hundred programmers I would guess in Hand, it took years to get something out the door. I would guess that some Projects/other developers would be happy to Profit from your Knowledge and abbilities. This also may Benefit your Health and Prevent from falling back in some Kind of burn out Syndrom.
All the best.

[RickCHodgin]

...FAT32 OS/2 partition for the OS/2 install...

OS/2 can boot from FAT, HPFS and JFS. I've never heard it can boot from FAT32.

Quite correct.  My mistake.  It's just FAT.  During installation you see the OS/2 format option.  Under file Windows disk management you see the disk partition info.  I had just assumed its "FAT" reference was FAT32.  Kind of freaky / surprising that it's not.

[RickCHodgin]

Once that is complete, I'll design OS/2 tools which will allow me to create a custom loader, and intercept and redirect all DLL references, allowing me to perform the equivalent of a Windows procmon.exe capture for any apps of interest.

To be honest: I do not understand why you try to reinvent the Wheel. I would like to Point you to the already existing, I would say THE OS/2 LOADER, where ArcaOS seems to base on as well, QSINIT. I see no reason to create another one. And, as others already stated, I would like to suggest that you take part in already existing OS/2 Projects, to concentrate on one Problem instead of writing everything from Scratch. Even for IBM with (I do not know exactly the number) some hundred programmers I would guess in Hand, it took years to get something out the door. I would guess that some Projects/other developers would be happy to Profit from your Knowledge and abbilities. This also may Benefit your Health and Prevent from falling back in some Kind of burn out Syndrom.
All the best.

Nearly everyone in my life tells me this.  But deep and permeating within my heart is a goal to create something which expressly and purposefully honors the flag and banner I pursue with my life.  It's not done for the project, or money, or fame.  It's done because I have these low-level development abilities, and an understanding of x86 hardware and OS design, and I wish to contribute something toward those ends.  But, I don't want to just do it to get it done.  I want to do it for that purpose I'm in pursuit of.

If you'd like to move to another forum I'll explain everything, but the bottom line is I'm not just a doer ... I'm a purposeful, conscientious doer, and I'm doing it for a real reason that is exceedingly strong within me.

[Dave Yeo]

FAT is probably a bad choice for an install volume due to its limits. 8.3 names (the WPS supports long names in extended attributes but the kernel can only see 8.3) along with the 2GB partition limit, large clusters and such. HPFS might be a good choice for you, there are DOS HPFS drivers if you need to access the OS/2 partition from DOS.
You also need an OS/2 aware boot manager if booting from anywhere but C: as OS/2 needs to know where it is installed. You can use the supplied OS/2 Bootmanager, it needs a primary partition of 1 cylinder.
As for fixpaks, they're installed after you get the system running. And doscall1.dll is matched to the kernel somewhat, you need a SMP doscall1.dll to go with the SMP kernel, which internally is a DLL and includes doscall0.

OS/2 also includes a lot of debugging stuff, including tools to trace stuff as you're describing. There's also sym files that you could disassemble to get the correct symbol names of various functions.
Good luck, it's a huge job you're undertaking.

[RickCHodgin]

FAT is probably a bad choice for an install volume due to its limits. 8.3 names (the WPS supports long names in extended attributes but the kernel can only see 8.3) along with the 2GB partition limit, large clusters and such. HPFS might be a good choice for you, there are DOS HPFS drivers if you need to access the OS/2 partition from DOS.
You also need an OS/2 aware boot manager if booting from anywhere but C: as OS/2 needs to know where it is installed. You can use the supplied OS/2 Bootmanager, it needs a primary partition of 1 cylinder.
As for fixpaks, they're installed after you get the system running. And doscall1.dll is matched to the kernel somewhat, you need a SMP doscall1.dll to go with the SMP kernel, which internally is a DLL and includes doscall0.

OS/2 also includes a lot of debugging stuff, including tools to trace stuff as you're describing. There's also sym files that you could disassemble to get the correct symbol names of various functions.
Good luck, it's a huge job you're undertaking.

I appreciate any help you offer, Dave.

[Dave Yeo]

I'd suggest installing OS/2 a couple of times in different ways, eg different partitions, different file systems and such just to get a feel for it. Installing in a FAT partition involves a different boot procedure then using an IFS (installable file system), installing on an extended partition means having a bootloader to start the boot process and pass on to the kernel where it is booting from so it can load its drivers. And so on.
You might also want to look at airboot https://trac.netlabs.org/air-boot a GPLv3 bootloader written in assembly and aimed at OS/2 which lives in the MBR area.

[RickCHodgin]

I'd suggest installing OS/2 a couple of times in different ways, eg different partitions, different file systems and such just to get a feel for it. Installing in a FAT partition involves a different boot procedure then using an IFS (installable file system), installing on an extended partition means having a bootloader to start the boot process and pass on to the kernel where it is booting from so it can load its drivers. And so on.
You might also want to look at airboot https://trac.netlabs.org/air-boot a GPLv3 bootloader written in assembly and aimed at OS/2 which lives in the MBR area.

I have installed it multiple times.  I have primarily used HPFS.  The reason I chose FAT this time was because I needed the ability to easily access the partition from other OSes (primarily DOS, Windows, and Linux).  I've never installed the HPFS driver for DOS.  Since I can do that I'll switch.

I'll work on installing the boot loader.  I used it back in the 90s with Warp 4 on my machine (DOS + OS/2 partitions).

For my own kernel, I developed code which allowed me to debug the boot sector code by putting the debugger code at known sequential locations on the disk, and then baking those into the boot loader so it just iterated to load Nn sectors continuously, and then did some interrupt descriptor table updates to handle INT1 and INT3.  I then issued INT3 to allow me to single-step through.  I only used bare bones BIOS functions, and it worked nicely.  That's the real mode debugger I mentioned previously.  I used this pb.ini (put boot) file to determine what to write, and wrote it to disk with this utility.  Not my best work, but it worked.

[Dave Yeo]

Various HPFS drivers for DOS here, http://hobbes.nmsu.edu/h-browse.php?dir=/pub/dos. Linux includes a HPFS read/write module though not all dists install it by default. I believe most Debian based dists do. NT used to be able to access HPFS through the pinball driver, capability probably went away after XP. Don't use it on partitions over 4.3GBs.

[RickCHodgin]

Various HPFS drivers for DOS here, http://hobbes.nmsu.edu/h-browse.php?dir=/pub/dos. Linux includes a HPFS read/write module though not all dists install it by default. I believe most Debian based dists do. NT used to be able to access HPFS through the pinball driver, capability probably went away after XP. Don't use it on partitions over 4.3GBs.

My real goal is the tracing.  If you can help me with any kernel debuggers, setting up the ability to record any traces, or other such features, that would be great.  I had intended to use the sym files to obtain meaningful names for the kernel version I will use.

[Dave Yeo]

Well there's the icat kernel debugger, http://hobbes.nmsu.edu/download/pub/os2/dev/util/icos0602.zip. I believe ideally you need 2 boxes with a serial connection to use it. Perhaps start here, http://home.earthlink.net/~steve53/os2diags/index.html for more info on using it.

[RickCHodgin]

Well there's the icat kernel debugger, http://hobbes.nmsu.edu/download/pub/os2/dev/util/icos0602.zip. I believe ideally you need 2 boxes with a serial connection to use it. Perhaps start here, http://home.earthlink.net/~steve53/os2diags/index.html for more info on using it.

Has anyone ever set it up with two VMs?

[Dave Yeo]

Don't know. Don't see why it wouldn't work.

[Lars]

Well there's the icat kernel debugger, http://hobbes.nmsu.edu/download/pub/os2/dev/util/icos0602.zip. I believe ideally you need 2 boxes with a serial connection to use it. Perhaps start here, http://home.earthlink.net/~steve53/os2diags/index.html for more info on using it.

Has anyone ever set it up with two VMs?

I am Using it from a VM but the target is a real OS/2 Machine. I am Using it With UDP but Then you cannot debug System Startup.

[Valery Sedletski]

2AndiB: OS/2 can boot from FAT32 (and from FAT/VFAT/exFAT too) with my FreeLDR bootloader. (and sysinstx command for fat32.ifs installs FreeLDR boot block on a medium). If someone interested, there is Team BOOT/2 boot disk, in which creation I participated. It is able to boot from FAT/FAT32/exFAT/HPFS/JFS/CDFS/etc. Can be installed on a LiveCD, as well as on a boot flash stick/USB harddisk/ATA harddisk/etc. It supports VirtualPC and VBox additions, so it is good for testing OS/2 in VM's. The latest version is here: ftp://osfree.org/upload/ramfsos2-20161021.zip

PS: 2Rick Hodgin: last OS/2 port of VBox I did supports redirecting the guest COM port to a named pipe. You can connect to this named pipe with DTerm terminal emulator (which supports listening the named pipes). This way, you can work with guest OS/2 kernel kernel debugger. So, you could debug the OS/2 kernel in VBox, and avoid using two real machines for debugging (a debugee machine and a terminal machine connected to its COM port with a null-modem cable)

PPS: DTerm can be downloaded from its author's page at http://os2acw.googlepages.com/. (Unfortunately, the author has been died, and no more new versions, and no sources. But it is a very good terminal emulator for kernel debugging).

PPPS: If you have VBox, but on Linux or Windows host, you can use miniterm terminal emulator on Linux, and Putty on Windows. You can choose "Host pipe" option here too. In Linux case, a local socket is used really, instead of a pipe. In Windows case, it uses a named pipe too. OS/2 pipes are specified as \pipe\dterm in both DTerm settings and VBox settings. Windows pipes are specified as \\.\pipe\somepipe. In Linux, sockets are created in /tmp. So, you specify /tmp/vboxfifo in VBox, and "unix#/tmp/vboxfifo" in minicom. In all three cases, it's working quite well.

PPPPS: And BTW, FAT is supported by fat32.ifs, including VFAT long file names. So, 8.3 limits are no more applicable. So, OS/2 sees the long file names on FAT without problems.

[RickCHodgin]

PS: 2Rick Hodgin: last OS/2 port of VBox I did supports redirecting the guest COM port to a named pipe. You can connect to this named pipe with DTerm terminal emulator (which supports listening the named pipes). This way, you can work with guest OS/2 kernel kernel debugger. So, you could debug the OS/2 kernel in VBox, and avoid using two real machines for debugging (a debugee machine and a terminal machine connected to its COM port with a null-modem cable)

PPS: DTerm can be downloaded from its author's page at http://os2acw.googlepages.com/. (Unfortunately, the author has been died, and no more new versions, and no sources. But it is a very good terminal emulator for kernel debugging).

PPPS: If you have VBox, but on Linux or Windows host, you can use miniterm terminal emulator on Linux, and Putty on Windows. You can choose "Host pipe" option here too. In Linux case, a local socket is used really, instead of a pipe. In Windows case, it uses a named pipe too. OS/2 pipes are specified as \pipe\dterm in both DTerm settings and VBox settings. Windows pipes are specified as \\.\pipe\somepipe. In Linux, sockets are created in /tmp. So, you specify /tmp/vboxfifo in VBox, and "unix#/tmp/vboxfifo" in minicom. In all three cases, it's working quite well.

I will need help getting that setup.  What I'm seeking to do is use external tools to examine and map the internal workings of the kernel and related system drivers / DLLs, and for the express purpose of finding out where it is subject to weaknesses or flaws resulting in potential data loss.