Author Topic: ES/1 open source kernel  (Read 3809 times)

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #15 on: November 25, 2017, 07:28:00 am »
ES/1 is getting some knowledge of file systems.  I'm working primarily on the ES/1 Disk Inspector tool.  It recognizes primary partitions with FAT12/16/32, HPFS, JFS, and a few other volume types, and it acknowledges the existence of extended partitions but does not parse them yet.

My next goal is to develop, from scratch, file system algorithms supporting sector-based FAT12 and FAT16 disks.  I will then move to an LBA-supported FAT16 volume.  Once these are written, they will be base drivers for ES/1, and will be used in ES/2 as well.

I plan to use those file systems in ES/1 for quite a while until I am able to get some additional tools written for kernel development.  Once I get to a certain point of a maturing toolset developed in ES/1, and remote debugging support added to my ES/2 kernel, I'll begin doing JFS development and, once that file system is completed, I will never look back. :-)

I've been working at putting together a comprehensive view of the OS/2 kernel's internal design as is visible from its API sitting atop i386 architecture.  This will help me determine what structures and mechanisms must be in place to support all of its needs.

Things are moving forward.  I'm doing this on all of my free evening, weekend, sick day, and holiday time periods.  Would like to do it full-time.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #16 on: November 25, 2017, 04:01:04 pm »
The latest Disk Inspector screen.  It has many added features, but some information is still parsed incorrectly due to subtleties between versions of the data structures.

I'll be working on a MOUNT command, which allows a partition to be mounted based on its partition and boot parameter block information.  Also, a CHKDISK command to cycle through all folders, and report on any anomalies.

These are all new algorithms.  I am not using any existing code, but only books and literature relating to the design of these data structures, along with existing examples on my own hard disks from other tools which created them properly (presumably). :-)

 

Andreas Kohl

  • Full Member
  • ***
  • Thank You
  • -Given: 6
  • -Receive: 9
  • Posts: 216
    • View Profile
    • warpserver.de
Re: ES/1 open source kernel
« Reply #17 on: November 25, 2017, 04:13:46 pm »
ES/1 is getting some knowledge of file systems.  I'm working primarily on the ES/1 Disk Inspector tool.  It recognizes primary partitions with FAT12/16/32, HPFS, JFS, and a few other volume types, and it acknowledges the existence of extended partitions but does not parse them yet.
HPFS and JFS are installable file systems and not "volume types". There exist only two volume types under OS/2's LVM: compatiblity and extended (that can consist of more than one partition). We can go to the past of OS/2's FDISK were your "volume types" shoul be some kind of partition id. But there was never a partition id for primary partitions that could be called HPFS. The specified ID for IFS partitions is used by different other file systems, e.g. NTFS. Fortunately DOS 3.x and all derived systems can handle IFS. But who cares...

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #18 on: November 25, 2017, 04:15:21 pm »
HPFS and JFS are installable file systems and not "volume types".

Disk volume types.  The entire partition is dedicated to a single identified file system type.

UPDATE:
Jan van Wijk's partition overview PDF
Video #1 with information on partitions plus DFSee (2014)
Video #2 with information on partitions plus DFSee (2016)
Video #3 with information on partitions plus DFSee (2016)
« Last Edit: November 25, 2017, 04:55:25 pm by Rick C. Hodgin »

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #19 on: November 25, 2017, 04:32:29 pm »
...there was never a partition id for primary partitions that could be called HPFS. The specified ID for IFS partitions is used by different other file systems, e.g. NTFS. Fortunately DOS 3.x and all derived systems can handle IFS. But who cares...

In my particular case, I happen to know that the partition is formatted HPFS.  And for the purposes of my test systems, it will be likewise, which is why it's identified as OS2HPFS for ID #7.

Adding algorithms to better differentiate between multiple occupants of the same identified partition ID is easy to do at some future point.  I'm working on base algorithms and key design points for those things ES/1 and ES/2 will support.  I'm not attempting to make a universal identifier.  I only support FAT12/16/32, HPFS, JFS, FAT16 Big, and "Other" right now.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #20 on: November 26, 2017, 10:01:32 am »
I am having dreams at night of developing ES/2.  Literally ... am problem-solving and designing the kernel in my sleep.  Very rare occurrence for me.  Only happened on other projects a couple times in my whole life, and none lately.

Andreas Kohl

  • Full Member
  • ***
  • Thank You
  • -Given: 6
  • -Receive: 9
  • Posts: 216
    • View Profile
    • warpserver.de
Re: ES/1 open source kernel
« Reply #21 on: November 27, 2017, 12:19:16 am »
In my particular case, I happen to know that the partition is formatted HPFS.  And for the purposes of my test systems, it will be likewise, which is why it's identified as OS2HPFS for ID #7.
To make a guess or suppose something doesn't qualify to know about it. With this incompatible scope you have to care about that prevents users from creating another filesystems. At least two points of failure and restriction on functionality.

Quote
Adding algorithms to better differentiate between multiple occupants of the same identified partition ID is easy to do at some future point.  I'm working on base algorithms and key design points for those things ES/1 and ES/2 will support.  I'm not attempting to make a universal identifier.  I only support FAT12/16/32, HPFS, JFS, FAT16 Big, and "Other" right now.
And what happens in the most simple case of a raw partition? And no support to recognise OS/2 Boot Manager?

Of course DFSee handles a limited set of widely used partition schemes under IBM PC AT compatible systems. Last time when I checked 3 or 4. But if you have also to deal with UNIX system divisions or similar stuff it's no longer useful. Later versions added at least some limited Mac support.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #22 on: November 27, 2017, 02:01:25 am »
To make a guess or suppose something doesn't qualify to know about it. With this incompatible scope you have to care about that prevents users from creating another filesystems. At least two points of failure and restriction on functionality.

And what happens in the most simple case of a raw partition? And no support to recognise OS/2 Boot Manager?

I'm in the algorithm design and test phase at this point on my own equipment.  This will not be for general use at this point.

Once completed it will become part of a driver which handles disks that have been identified by another layer as being a particular type.  It will then route requests of these types to those drivers.
« Last Edit: November 27, 2017, 02:08:16 am by Rick C. Hodgin »

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #23 on: November 29, 2017, 02:46:56 pm »
I am working on the keyboard mapper.  I am intercepting INT 9, and recording actually make and break codes for each physical key on the keyboard.  These then go through a mapper which identifies the physical keyboard layout, and through a mapper to a language application for the keyboard characters.

For now ES/1 will only know and support US English, but the capabilities are there to create one for any language.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #24 on: December 03, 2017, 07:57:10 am »
I am working on using graphics mode rather than text mode.  I'll only support VBE modes for now (VESA modes available through BIOS calls).  I'm looking to support apps with graphics and/or text with ES/1's first release.  I plan to have (James 4:15) text-rendering algorithms for the graphics window, and some primitive window abilities.

Once I get the graphics algorithms working, I'll return to the keyboard mapper as I want to create a graphics app to visualize how it works.  After the keyboard is setup I'll focus on the mouse.  After that, it will be all hard disk algorithms for a basic FAT16 partition.  At that point, I'll have something to demonstrate to people.  They'll be able to download the system, install it on a virtual or real machine, and experiment with using it.

My goal after that is compiler and assembler support.  Once those are up and running, I'll begin development of the ES/2 kernel and create a full debugger ability to walk from real-mode code into protected mode code, and to be able to have all aspects debugged.

UPDATE:  If interested, you can download all of the source code here:
Code: [Select]
git clone http://www.libsf.org:8990/scm/lib/libsf.git
« Last Edit: December 03, 2017, 02:24:39 pm by Rick C. Hodgin »

Sergey Posokhov

  • Full Member
  • ***
  • Thank You
  • -Given: 1
  • -Receive: 5
  • Posts: 107
    • View Profile
    • OS/2 API Research
Re: ES/1 open source kernel
« Reply #25 on: December 03, 2017, 04:09:05 pm »
Then Intel will completely drop BIOS support on their motherboards and CPUs.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #26 on: December 03, 2017, 04:18:19 pm »
Then Intel will completely drop BIOS support on their motherboards and CPUs.

If so we can work around it.  We just need to come together and work for a common goal.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #27 on: December 04, 2017, 05:06:44 am »
I have ES/1 writing to the screen in graphics mode.  The 3dfx voodoo3 2000 card I'm using does 132x60 text natively (132x80 if you load an 8x6 font), but graphics-wise it is late 90s tech and only goes up to 1280x1024x24 colors, but it's enough for testing.  Even in that mode I'm using an 8x14 font and am able to get a 160x43 console window.  I have to use "bigreal" or "unreal" mode to access the frame buffer to write data (which is far beyond 1 MB in memory -- this particular card is using 0xfa000000 for the start of the buffer), which means everything is running through some extra assembly steps to populate the on-screen content.  It makes it pretty slow right now (about 1 second to write content to the entire screen) using my 2.2 GHz AMD64-based CPU.  Once I get the graphics engine working decently, I can switch to my more modern video card and use higher resolutions at faster speeds on its faster bus (the 3dfx card is just PCI).

ES/1 should support general purpose graphics, including a desktop, mouse pointer, flashing caret (cursor where you type), multiple text-based console windows, a mini-Presentation Manager for real graphical apps, and be able to run several separate programs simultaneously.

I'll be using my own design "dynamic scalable font" (DSF) engine for all fonts.  I plan to also have a setup tool to allow you to create your own fonts using its design (James 4:15 -- Lord willing).  My first fonts are libsf.dsf and libsf_mono.dsf, which are similar to Ubuntu's font design, but a little less bold, narrower, and with some character changes.

I'm fairly happy with how things are progressing.  It's slow and steady work.  I hope to have an ES/1 kernel booting with a pre-alpha static image in Jan.2018.  It would be developed enough to allow different people to see if it boots, has a mouse pointer, does graphics, responds to mouse, keyboard, and timer events, etc, so I could get some feedback.
« Last Edit: December 04, 2017, 05:12:34 am by Rick C. Hodgin »

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #28 on: December 04, 2017, 02:23:08 pm »
If any of you want to follow along with my development and test out what I'm doing on your local machine, do these steps:

Code: [Select]
(1)  Install IBM PC DOS 7 (or other DOS system) on one of your partitions
(2)  Install Open Watcom for DOS
(3)  Clone my libsf repository to your local C:\ drive in a Windows machine.
     Note:  It has more than just ES/1 and ES/2 in it, and it uses long filenames
(4)  Copy the \libsf\es2\ES1\ and \libsf\es2\ES2\ to your DOS partition's C:\ES1\ and C:\ES2\ folders.
     Note:  I always use a "g.bat" file and a "d.bat" in the current director for "go compile" and "debug".
     Note:  You can type "g" to begin compiling, and "d" to begin debugging.
     Note:  You'll need to setup your environment to C:\WATCOM\ or wherever you install Open Watcom.
     Note:  I will post my SETENV.BAT content when I get back on that machine.

If you don't have a copy of IBM PC DOS 7, it can still be found on places like eBay for sale.  IBM PC DOS is also available for download online.  MS-DOS will probably work as well as other versions of DOS.  It doesn't use anything fancy, though some versions of the Watcom compiler use DPMI and DOS4GW.

Download Open Watcom for DOS.  Note:  You'll want the "open-watcom-c-dos-1.9.exe" file for download, and you'll probably want to rename it "winstall.exe" for use in DOS.  Download it in Windows and copy to your DOS partition so when you reboot it's there.

The git repository to clone is here:
Code: [Select]
C:\> git clone http://www.libsf.org:8990/scm/lib/libsf.git
Cloning into libsf...

The folders to copy are here.  On your local hard-drive they will appear here:
Code: [Select]
C:\libsf\> CD ES2
C:\libsf\ES2\> DIR /A:D
 Volume in drive C is LABEL
 Volume Serial Number is 1234-5678

 Directory of c:\libsf\es2

11/19/2017  11:17 PM    <DIR>          .
11/19/2017  11:17 PM    <DIR>          ..
12/03/2017  12:57 AM    <DIR>          ES1
11/12/2017  06:51 PM    <DIR>          ES2
               0 File(s)              0 bytes
               4 Dir(s)   1,310,162,944 bytes free

You'll need to use Windows Explorer or execute these commands to copy it over (assuming D: is your DOS partition, but replace D: with whatever it is):
Code: [Select]
X:\> C:
C:\> CD \LIBSF\ES2\
C:\libsf\es2\> XCOPY /S ES*\*.* D:\*.*

In the end you'll want to have a DOS partition which has at least these folders:
Code: [Select]
C:\> DIR /A:D
 Volume in drive C is LABEL
 Volume Serial Number is 1234-5678

 Directory of c:\

11/19/2017  11:17 PM    <DIR>          .
11/19/2017  11:17 PM    <DIR>          ..
11/12/2017  06:51 PM    <DIR>          DOS
12/03/2017  12:57 AM    <DIR>          ES1     <=== ES/1 development
11/12/2017  06:51 PM    <DIR>          ES2     <=== ES/2 development
11/12/2017  06:51 PM    <DIR>          WATCOM
               0 File(s)              0 bytes
               6 Dir(s)   1,310,162,944 bytes free

Once there you can navigate to "\es1\disk\" and type "g.bat".  It probably won't work properly on your machine because it's hard-coded right now to the 3dfx voodoo3 2000 and VESA mode 0x11b is my 1280x1024x24 mode.  But, I will soon be removing the hard-codings and you can use the software.

In the mean-time you can examine the source code.  The main files are:
Code: [Select]
\es1\vbe\*.*
\es1\vio\*.*
\es1\common\*.*
\es1\disk\*.*
\es1\keyflags\*.*

I'm using The SemWare Editor Jr. for DOS to do my editing, though I actually have an old version from the 1990s before the author made The SemWare Editor for Windows, so mine is actually just called The SemWare Editor.  It was written by a man name Sammy Edward Mitchell ("SEM" in SemWare), from Marietta, Georgia back in the late 80s.  It used to be sold under the name "QEdit" in the BBS days.

-----
If any of you want to start up a dialogue with me and ask questions about development or how to do something, you can reach me at my email address.  It is rick.c.hodgin and I use GMail.  If you can put those together you'll be able to reach me. :-)
« Last Edit: December 04, 2017, 02:59:44 pm by Rick C. Hodgin »

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 17
  • -Receive: 2
  • Posts: 246
    • View Profile
    • Liberty Software Foundation
Re: ES/1 open source kernel
« Reply #29 on: December 05, 2017, 02:50:09 pm »
I plan on creating a real-mode emulator that will run in 32-bit protected mode, allowing for all existing 16-bit code to run natively in ES/2 through emulation.  Drivers can be supported, but not the full BIOS machine state.  We'll address each limiting factor one-by-one as they come up, and will support everything necessary to make all existing OS/2 drivers work, but it won't be a full machine emulation.  Just enough to make everything that currently exists work properly.

I will begin development on the 16-bit emulator as a later part of ES/1 development.  It will be called Em86+, and will serve as one of the assets available to ES/2 in moving forward.

I also plan a full flat 32-bit 386-like ISA emulator (an 80386 environment with some ISA extensions, called Em386+) which will allow code to be written for a virtual 386+ machine that can be suspended, and resumed at any state.  It will not run at native machine speed but in emulation at probably 30-50% machine speed, but it would be able to run as required, in a known environment, and on every type of hardware natively.

Em386+ will be a much simpler and somewhat extended 80386 environment, with additional instructions, 3x the number of registers, and a flat address space beginning at offset 0 and continuing up.  The ES/1 and ES/2 compilers will be able to write code which targets this environment.

One advantage of using Em86+ and Em386+ will be when there is an error, the machine state can be saved at that very point.  You can also enable debug traces to diagnose the last several thousand instructions to figure out what sequence of events allowed something to arrive at an error state.

I eventually plan to incorporate these new 86+ and 386+ environments into my Arxoda CPU design, allowing them to eventually run on native hardware at full machine speed.  They will also include debug trace abilities.  Long term goals.
« Last Edit: December 05, 2017, 03:04:43 pm by Rick C. Hodgin »