Author Topic: Custom OS/2 Kernel Debugger  (Read 726 times)

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Custom OS/2 Kernel Debugger
« on: November 24, 2017, 03:04:22 pm »
Does anybody know if there are kernel APIs which would allow a custom OS/2 debugger driver/PSD to be created?  And if so, are there any open source examples I could look at?



Andreas Kohl

  • Full Member
  • ***
  • Thank You
  • -Given: 6
  • -Receive: 9
  • Posts: 216
    • View Profile
    • warpserver.de
Re: Custom OS/2 Kernel Debugger
« Reply #1 on: November 24, 2017, 04:34:06 pm »
Does anybody know if there are kernel APIs which would allow a custom OS/2 debugger driver/PSD to be created?
PSDs were introduced with OS/2 for SMP and the corresponding Toolkit add-on should have some information about it. I'm quite sure that this information was updated in later Warp Server SMP releases. An example PSD should be contained inside of DC DDK.
The Kernel API supports DosPTrace() for older 16-bit debuggers and since Version 2.0 the 32-bit DosDebug().

I recommend to read "OS/2 2.11, OS/2 2.1 for Windows, New OS/2 Device Drivers, and Updates to OS/2 2.1" (Document Number GG24-4212-00) which refers to further information on specific topics and gives a good overview.

[/quote]And if so, are there any open source examples I could look at?[/quote]
There was (or still is) a project targetting the L4 micro kernel, similar to the unlucky Warp PPC (wpos).

Lars

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 21
  • Posts: 166
    • View Profile
Re: Custom OS/2 Kernel Debugger
« Reply #2 on: November 24, 2017, 04:53:33 pm »
Does anybody know if there are kernel APIs which would allow a custom OS/2 debugger driver/PSD to be created?  And if so, are there any open source examples I could look at?

There is a more or less undocumented Device Helper (DevHlp_RegisterKDD, function code 0x83) that allows a remote debugger to send commands to the debug kernel and receive the returns and also events. For example, this Device Helper is used by KDBNET.SYS to support debugging via UDP network connection.
The "Control Programming Guide and Reference" contains a chapter "Kernel Debugger Communications Protocol" that sheds some light on the commands you can send and also the returns and events you will receive.

At one point in time I dug into this stuff to find out what registers are used in the Device Helper and what format the input and output buffers have to be. Unfortunately I lost my notes (they were incomplete anyway).

You'll need to delve into KDBNET.SYS and do some reverse engineering.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: Custom OS/2 Kernel Debugger
« Reply #3 on: November 24, 2017, 05:09:05 pm »
At one point in time I dug into this stuff to find out what registers are used in the Device Helper and what format the input and output buffers have to be. Unfortunately I lost my notes (they were incomplete anyway).

You'll need to delve into KDBNET.SYS and do some reverse engineering.

I was impressed to read about your big floppy driver for USB disks, and I was impressed by David Azewericz's 2015 Warpstock videos on device drivers.  It got me thinking about the possibility of writing a kernel plug-in which allows us to write our own debugger with custom abilities to reveal and probe kernel aspects.

I've contacted someone with strong IBM contacts with the hope of soon getting legal license to decompile and reverse-engineer the kernel for research purposes in understanding the breadth of its current design for replacement and extension in ES/2.

Lars

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 21
  • Posts: 166
    • View Profile
Re: Custom OS/2 Kernel Debugger
« Reply #4 on: November 24, 2017, 07:10:43 pm »
Thanks.

Ploughing through my pile of paper I found the notes:

dh = RegisterKDD (0x83)
eax = Command Code (1 = valid value)
es:di control buffer address (4 DWORDS)
DWORDS in control buffer:
1. DWORD: ???
2. DWORD: lower WORD: IRQ number (what IRQ ?)
3. DWORD far pointer to code -> Handler routine (into the code segment of the invoking driver)
4. DWORD: far pointer to buffer (into the data segment of the invoking driver)

Control Buffer is then copied to symbol address _KDDC (segment "c" of DOSCALLS at offset 1233c)

USHORT FAR HandlerRoutine(USHORT arg1=ax, USHORT arg2=cx)
arg1 = function code 1,2,3
arg2 = KDB function ?

The Device Helper "RegisterKDD"  (dh_RegisterKDD) can be found at 3:1528 (3. segment of DOSCALLS module, offset 1528) in the 14.104a SMP debug kernel (the latest available apart from what AN offers)
You can use Theseus disassembly to have a look into how it operates. You can also use Theseus module view to find out at what target address the various DOSCALLS segments are loaded.
Of course, the retail kernel contains no operational code (even though it also exports symbol dh_RegisterKDD) it only returns with the carry flag set to indicate an error:

push ds
mov eax,5
stc
pop ds
retn


I remember there was some way to find out which driver registered with the KDB (that's how I found out that KDBNET.SYS indeed calls this Device Helper to register with the debug kernel) but I cannot remember how I found out. I guess I looked at symbol _KDDC ...
« Last Edit: November 24, 2017, 07:25:27 pm by Lars »

Andreas Kohl

  • Full Member
  • ***
  • Thank You
  • -Given: 6
  • -Receive: 9
  • Posts: 216
    • View Profile
    • warpserver.de
Re: Custom OS/2 Kernel Debugger
« Reply #5 on: November 24, 2017, 10:26:16 pm »
There is a more or less undocumented Device Helper (DevHlp_RegisterKDD, function code 0x83)
At least it's mentioned in the Programming Addendum of the later OS/2 Toolkits. But they missed to dedicate a section for it.

Quote
that allows a remote debugger to send commands to the debug kernel and receive the returns and also events. For example, this Device Helper is used by KDBNET.SYS to support debugging via UDP network connection.
Of course the OS/2 KDB NET Driver was introduced with the LAPS included with TCP/IP 4.1. It's some kind of embedded terminal server functionality. It removes the need of a physical terminal server and it's cabling. And it works quite reliable in TR or switched Ethernet segments. But requires matching debug kernels.

Quote
You'll need to delve into KDBNET.SYS and do some reverse engineering.
Maybe it should mentioned that IBM's Application/System Debug Tool (ASDT32) is sometimes helpful for debugging device drivers and free available from SDP tools or already included with the DDK.

Watcom's C/C++ also shipped with remote debugging support. It could be worth checking the source tree of OpenWatcom for surviving parts.

Lars

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 21
  • Posts: 166
    • View Profile
Re: Custom OS/2 Kernel Debugger
« Reply #6 on: November 25, 2017, 09:19:05 am »
Of course the OS/2 KDB NET Driver was introduced with the LAPS included with TCP/IP 4.1. It's some kind of embedded terminal server functionality. It removes the need of a physical terminal server and it's cabling. And it works quite reliable in TR or switched Ethernet segments. But requires matching debug kernels.

I don't see a need for some match of debug kernel version. The driver (KDBNET.SYS) just uses the Dev Help to register itself with the kernel. From then on the kernel will know which driver to call and what address to pass debugging data to.

Andreas Kohl

  • Full Member
  • ***
  • Thank You
  • -Given: 6
  • -Receive: 9
  • Posts: 216
    • View Profile
    • warpserver.de
Re: Custom OS/2 Kernel Debugger
« Reply #7 on: November 25, 2017, 05:49:48 pm »
I don't see a need for some match of debug kernel version. The driver (KDBNET.SYS) just uses the Dev Help to register itself with the kernel. From then on the kernel will know which driver to call and what address to pass debugging data to.
Although I cannot confirm that kdbnet.sys works with every OS/2 debug kernel, so "matching" should be replaced by "not too ancient".