OS2 World Community Forum

Public Discussions => General Discussion => Topic started by: RickCHodgin on November 16, 2017, 09:37:28 pm

Title: ES/1 open source kernel
Post by: RickCHodgin on November 16, 2017, 09:37:28 pm
Quote from: Rick C. Hodgin
Nov.13.2017 -- A quick update:

I've begun coding a DOS-like ES/1 operating system as a precursor to my ES/2 kernel.  It will be a simple text-based kernel in real-mode allowing me to port compilers, assemblers, linkers, debuggers, to that system, allowing me to test code in a simple environment, one I can easily mimic in IBM PC DOS 7.

ES/1 will also be released as a simple public OS people will be able to download and use for FAT12 and FAT16 disks.  It will also natively allow serial port networks. :-)

Quote from: Rick C. Hodgin
Nov.16.2017 -- I have a text-based forms mechanism.  I'm currently developing my own master boot record, partition, FAT12 and FAT16 drivers, as well as an ES/1 compiler, assembler, linker, and debugger.  A large portion of that code is being ported from my existing kernel code.

I still have to complete:  COM port I/O, the basic ES/1 command prompt.

I will send screenshots at some point.  I'm using a 3dfx Voodoo3 2000 card as it does a 132x60 text mode (at 0xb8000 for those who know what that means, allowing me to reclaim 0xa0000 - 0xb7fff for real mode memory).

ES/1 will boot like a basic 16-bit DOS running on an 80386 or higher, and have a developer environment.  It will be a springboard for 16-bit ES/2 development.

It will boot up as:
Code: [Select]
ES/1 Operating System - Command Prompt
Nov.16.2017 06:30 am

[c:\es1\] _

I've created this thread to discuss ES/1 development and to answer any questions.  ES/1 is basically a stand-alone self-bootable version of what will become the 16-bit components present in ES/2:

My goals:

Note also that ES/1 will have a nice set of basic abilities.  It will serve as a decent launching pad for any custom kernel efforts.  Since ES/1 will pretty much use only what BIOS has already established, you can load your kernel into 0x0000:0x7c00 and single-step debug your way through all real-mode code.  If you write it in my compiler and/or assembler, you'll have source-level debugging abilities to boot your own kernel.

--
Rick C. Hodgin
Title: Re: ES/1 open source kernel
Post by: ak120 on November 16, 2017, 09:43:46 pm
So how it runs inside a VDM?
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 16, 2017, 09:53:54 pm
So how it runs inside a VDM?

I'm running it on native hardware, but it should run perfectly fine as it uses very basic BIOS services for keyboard, mouse, disk, video, timer, with pretty much everything else being completely custom developed.  It won't have any USB support, save any BIOS emulation of USB devices as standard devices (PS/2 keyboard + mouse, hard drives, etc).

I will use it to develop the ES/1 and ES/2 networking stack, and also the algorithms required for FAT32 and JFS file support.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 17, 2017, 02:40:35 pm
The user interface is becoming mature.  It will be integrated as a standard set of ES/1 DLLs available for linking against at runtime.  I'll try and post some screenshots this weekend.

My next goal is to get the disk services working.  These are internal support algorithms for these command line functions:

Code: [Select]
DIR {/w /a: /p}      // Normal directory
CD target            // Change directory
MD target            // Make directory
RD target            // Remove directory
COPY src dst         // Copy one file to another
COPY CON target      // Copy console content to file
TYPE {/hex} target   // Display file content
program              // Load an ES/1 program, .BAT, or .REX into memory
DEBI program         // Load an ES/1 program into memory and launch the Debi debugger
DATE                 // System date
TIME                 // System time
REBOOT {/cold}       // Restart ES/1 either warm or cold

At that point, the system will be usable (provided there are programs written in ES/1 to run).  After that, the compiler, assembler, linker, and debugger.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 18, 2017, 08:20:16 pm
So how it runs inside a VDM?

I was able to verify it runs in dosbox 0.74 for Windows.  I haven't been able to figure out how to get it to use larger than an 80x25 text screen in dosbox, however, so it's limited.  VESA reports no extended 132xYY text modes, so it may not be possible.  The things I've been designing have been for my personal use and use 132x60 video modes.  I'm working on posting some screen shots right now.
Title: Re: ES/1 open source kernel
Post by: ak120 on November 18, 2017, 10:01:10 pm
So how it runs inside a VDM?

I was able to verify it runs in dosbox 0.74 for Windows.  I haven't been able to figure out how to get it to use larger than an 80x25 text screen in dosbox, however, so it's limited.  VESA reports no extended 132xYY text modes, so it may not be possible.  The things I've been designing have been for my personal use and use 132x60 video modes.  I'm working on posting some screen shots right now.
DOSbox is a special case. It doesn't offer accurate emulation of an IBM PC compatible architecture. Fortunately even under Microsoft's OS/2 derived products there is a broader choice of emulation methods. Virtual PC or Virtual Server can do the trick for a limited set of modes. VirtualBox or QEMU use the Bochs Video BIOS. And PCem https://pcem-emulator.co.uk/status.html (https://pcem-emulator.co.uk/status.html) offers support for different graphic adapters.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 18, 2017, 10:12:23 pm
So how it runs inside a VDM?

I was able to verify it runs in dosbox 0.74 for Windows.  I haven't been able to figure out how to get it to use larger than an 80x25 text screen in dosbox, however, so it's limited.  VESA reports no extended 132xYY text modes, so it may not be possible.  The things I've been designing have been for my personal use and use 132x60 video modes.  I'm working on posting some screen shots right now.
DOSbox is a special case. It doesn't offer accurate emulation of an IBM PC compatible architecture. Fortunately even under Microsoft's OS/2 derived products there is a broader choice of emulation methods. Virtual PC or Virtual Server can do the trick for a limited set of modes. VirtualBox or QEMU use the Bochs Video BIOS. And PCem https://pcem-emulator.co.uk/status.html (https://pcem-emulator.co.uk/status.html) offers support for different graphic adapters.

I typically use VirtualBox and Bochs, but I haven't setup an IBM PC DOS 7 machine yet.  But I'm sure it will work because I only use the absolute smallest set of BIOS functions, the rest is all reliant only upon the hardware itself, and should work on any compliant CPU.

ES/1 in production will probably only be an 80x25 video system like normal consoles, and I'll probably allow 80x50, and other modes via VESA calls which can enumerate their capabilities (the 3dfx Voodoo3 2000, for example, enumerates 132x25, 132x43, 132x50, and 132x60 as text modes).

And, it will be open source so people can port it to whatever they like.  If the patch is good code, I'll accept it back upstream into the ES/1 mainline. :-)

I almost have the screen shot converter written.  I am capturing text with attributes to a disk file, and then running it through a post-processor to generate the .BMP file, which will be converted to a .PNG file.  Eventually I'll incorporate this ability in the text-based console using the iVio_screenCapture(filename); function.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 18, 2017, 11:55:40 pm
I doubt anyone cares about this project at this point, but to keep people posted here's where I am:  I had to use my ES/2 fonts for generating this image, and they use my specially mapped upper ASCII characters.  As such, they come through like the attached image, showing some characters incorrectly.  The characters around the double-lines look correct, but the single-line borders were usurped by my own font characters, so they don't show up properly in this image.

I'll have to download the original VGA font bitmaps from the graphics card and re-generate the image.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 19, 2017, 01:31:10 am
Updated image using the VGA 8x16 font.  The original image above was with an 8x14 font.

Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 19, 2017, 04:23:57 am
The two screens now rendered in an 8x14 font.  The navigation from highlighted blue/cyan window to each of the other windows is with F6 and Shift-F6.  Tab and Shift-Tab navigates input fields within each window.  F12 saves a screenshot.  Escape exits.  All keys and mouse movements are routed through custom callback functions.

Other such windows can be created.  A main form is created, and then windows are created and arranged within it.  And then objects are added to each window allowing for inputs like labels, textboxes, editboxes, checkboxes, and other controls can be added as needed.

The source code (http://www.libsf.org:8990/projects/LIB/repos/libsf/commits/5d9152d26abab6bafdab405b97d34a5f2c91e6e9) for rendering the screenshots has been checked in.  You can also see the font data extracted from the VGA card for 8x8, 8x14, and 8x16 modes.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 20, 2017, 03:24:16 am
I have the Disk Inspector reading data.  It also has checkboxes now, and the input boxes for the text and numbers are complete.  Still have date and time to complete.

Here we see the first disk sector on drive C:, which is device 128 (0x80 hex).
Title: Re: ES/1 open source kernel
Post by: ak120 on November 20, 2017, 12:18:26 pm
If you're keen on some publicity or a wider audience why not simply setting up a demo image with Jeff's PCjs (https://www.pcjs.org/). Just my two cents...

Et ne nos inducas in tentationem.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 20, 2017, 12:35:27 pm
If you're keen on some publicity or a wider audience why not simply setting up a demo image with Jeff's PCjs (https://www.pcjs.org/). Just my two cents...

Et ne nos inducas in tentationem.

I don't understand.  But, I'm not keen on publicity.  I have a lot of work ahead of me.  Until I get a bootable, semi-usable product, I would only post my updates here.  And if people aren't interested, I'm content to work without posting updates.

Here's my source code (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/ES1/VIO) for the screenshots above, the Disk Inspector (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/ES1/DISK), and Debi Debugger (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/ES1/DEBI).  It's designed for 132x60 text-based video modes.  Here's the utility which does the text screenshot converter (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/utils/text_screenshot).  And here are all the video algorithms (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/ES1/VIO/VIO.H).

All released under a PBL license (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/pbl_v1.txt).
Title: Re: ES/1 open source kernel
Post by: ak120 on November 20, 2017, 01:01:16 pm
I will not judge about the expedience of using C++ in your approach. But I can give the "wise" advise: better using C. Everything else can be called a lèse-majesté.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 20, 2017, 01:08:21 pm
I will not judge about the expedience of using C++ in your approach. But I can give the "wise" advise: better using C. Everything else can be called a lèse-majesté.

It is C code.  It uses a handful of C++ syntax relaxations, but with minor tweaks it will compile in a C compiler.

C++ doesn't require "struct" keyword, and allows anonymous unions and certain line comments.  And I also use parameter initialization, and if it came up I would use function overloading.  That's about all I use of C++ though.  I don't use the class, or exception handling, etc.  I just like to use the C++ compiler because it's like "an extended and somewhat simplified C," but it also adds additional type checking, which I also prefer.

UPDATE:  I thought of one more thing I use:  bool types. :-)
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 25, 2017, 07:28:00 am
ES/1 is getting some knowledge of file systems.  I'm working primarily on the ES/1 Disk Inspector tool.  It recognizes primary partitions with FAT12/16/32, HPFS, JFS, and a few other volume types, and it acknowledges the existence of extended partitions but does not parse them yet.

My next goal is to develop, from scratch, file system algorithms supporting sector-based FAT12 and FAT16 disks.  I will then move to an LBA-supported FAT16 volume.  Once these are written, they will be base drivers for ES/1, and will be used in ES/2 as well.

I plan to use those file systems in ES/1 for quite a while until I am able to get some additional tools written for kernel development.  Once I get to a certain point of a maturing toolset developed in ES/1, and remote debugging support added to my ES/2 kernel, I'll begin doing JFS development and, once that file system is completed, I will never look back. :-)

I've been working at putting together a comprehensive view of the OS/2 kernel's internal design as is visible from its API sitting atop i386 architecture.  This will help me determine what structures and mechanisms must be in place to support all of its needs.

Things are moving forward.  I'm doing this on all of my free evening, weekend, sick day, and holiday time periods.  Would like to do it full-time.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 25, 2017, 04:01:04 pm
The latest Disk Inspector screen.  It has many added features, but some information is still parsed incorrectly due to subtleties between versions of the data structures.

I'll be working on a MOUNT command, which allows a partition to be mounted based on its partition and boot parameter block information.  Also, a CHKDISK command to cycle through all folders, and report on any anomalies.

These are all new algorithms.  I am not using any existing code, but only books and literature relating to the design of these data structures, along with existing examples on my own hard disks from other tools which created them properly (presumably). :-)

 
Title: Re: ES/1 open source kernel
Post by: ak120 on November 25, 2017, 04:13:46 pm
ES/1 is getting some knowledge of file systems.  I'm working primarily on the ES/1 Disk Inspector tool.  It recognizes primary partitions with FAT12/16/32, HPFS, JFS, and a few other volume types, and it acknowledges the existence of extended partitions but does not parse them yet.
HPFS and JFS are installable file systems and not "volume types". There exist only two volume types under OS/2's LVM: compatiblity and extended (that can consist of more than one partition). We can go to the past of OS/2's FDISK were your "volume types" shoul be some kind of partition id. But there was never a partition id for primary partitions that could be called HPFS. The specified ID for IFS partitions is used by different other file systems, e.g. NTFS. Fortunately DOS 3.x and all derived systems can handle IFS. But who cares...
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 25, 2017, 04:15:21 pm
HPFS and JFS are installable file systems and not "volume types".

Disk volume types.  The entire partition is dedicated to a single identified file system type.

UPDATE:
Jan van Wijk's partition overview PDF (https://www.dfsee.com/present/pcpartit/pcpartit.pdf)
Video #1 (https://www.youtube.com/watch?v=UEMK4Nuzc2A) with information on partitions plus DFSee (2014)
Video #2 (https://www.youtube.com/watch?v=MwOqLX3JyT0) with information on partitions plus DFSee (2016)
Video #3 (https://www.youtube.com/watch?v=POqP9M9SZcE) with information on partitions plus DFSee (2016)
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 25, 2017, 04:32:29 pm
...there was never a partition id for primary partitions that could be called HPFS. The specified ID for IFS partitions is used by different other file systems, e.g. NTFS. Fortunately DOS 3.x and all derived systems can handle IFS. But who cares...

In my particular case, I happen to know that the partition is formatted HPFS.  And for the purposes of my test systems, it will be likewise, which is why it's identified as OS2HPFS for ID #7.

Adding algorithms to better differentiate between multiple occupants of the same identified partition ID is easy to do at some future point.  I'm working on base algorithms and key design points for those things ES/1 and ES/2 will support.  I'm not attempting to make a universal identifier.  I only support FAT12/16/32, HPFS, JFS, FAT16 Big, and "Other" right now.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 26, 2017, 10:01:32 am
I am having dreams at night of developing ES/2.  Literally ... am problem-solving and designing the kernel in my sleep.  Very rare occurrence for me.  Only happened on other projects a couple times in my whole life, and none lately.
Title: Re: ES/1 open source kernel
Post by: ak120 on November 27, 2017, 12:19:16 am
In my particular case, I happen to know that the partition is formatted HPFS.  And for the purposes of my test systems, it will be likewise, which is why it's identified as OS2HPFS for ID #7.
To make a guess or suppose something doesn't qualify to know about it. With this incompatible scope you have to care about that prevents users from creating another filesystems. At least two points of failure and restriction on functionality.

Quote
Adding algorithms to better differentiate between multiple occupants of the same identified partition ID is easy to do at some future point.  I'm working on base algorithms and key design points for those things ES/1 and ES/2 will support.  I'm not attempting to make a universal identifier.  I only support FAT12/16/32, HPFS, JFS, FAT16 Big, and "Other" right now.
And what happens in the most simple case of a raw partition? And no support to recognise OS/2 Boot Manager?

Of course DFSee handles a limited set of widely used partition schemes under IBM PC AT compatible systems. Last time when I checked 3 or 4. But if you have also to deal with UNIX system divisions or similar stuff it's no longer useful. Later versions added at least some limited Mac support.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 27, 2017, 02:01:25 am
To make a guess or suppose something doesn't qualify to know about it. With this incompatible scope you have to care about that prevents users from creating another filesystems. At least two points of failure and restriction on functionality.

And what happens in the most simple case of a raw partition? And no support to recognise OS/2 Boot Manager?

I'm in the algorithm design and test phase at this point on my own equipment.  This will not be for general use at this point.

Once completed it will become part of a driver which handles disks that have been identified by another layer as being a particular type.  It will then route requests of these types to those drivers.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 29, 2017, 02:46:56 pm
I am working on the keyboard mapper.  I am intercepting INT 9, and recording actually make and break codes for each physical key on the keyboard.  These then go through a mapper which identifies the physical keyboard layout, and through a mapper to a language application for the keyboard characters.

For now ES/1 will only know and support US English, but the capabilities are there to create one for any language.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 03, 2017, 07:57:10 am
I am working on using graphics mode rather than text mode.  I'll only support VBE modes for now (VESA modes available through BIOS calls).  I'm looking to support apps with graphics and/or text with ES/1's first release.  I plan to have (James 4:15) text-rendering algorithms for the graphics window, and some primitive window abilities.

Once I get the graphics algorithms working, I'll return to the keyboard mapper as I want to create a graphics app to visualize how it works.  After the keyboard is setup I'll focus on the mouse.  After that, it will be all hard disk algorithms for a basic FAT16 partition.  At that point, I'll have something to demonstrate to people.  They'll be able to download the system, install it on a virtual or real machine, and experiment with using it.

My goal after that is compiler and assembler support.  Once those are up and running, I'll begin development of the ES/2 kernel and create a full debugger ability to walk from real-mode code into protected mode code, and to be able to have all aspects debugged.

UPDATE:  If interested, you can download all of the source code here (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/ES1):
Code: [Select]
git clone http://www.libsf.org:8990/scm/lib/libsf.git
Title: Re: ES/1 open source kernel
Post by: Sergey Posokhov on December 03, 2017, 04:09:05 pm
Then Intel will completely drop BIOS support on their motherboards and CPUs.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 03, 2017, 04:18:19 pm
Then Intel will completely drop BIOS support on their motherboards and CPUs.

If so we can work around it.  We just need to come together and work for a common goal.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 04, 2017, 05:06:44 am
I have ES/1 writing to the screen in graphics mode.  The 3dfx voodoo3 2000 card I'm using does 132x60 text natively (132x80 if you load an 8x6 font), but graphics-wise it is late 90s tech and only goes up to 1280x1024x24 colors, but it's enough for testing.  Even in that mode I'm using an 8x14 font and am able to get a 160x43 console window.  I have to use "bigreal" or "unreal" mode to access the frame buffer to write data (which is far beyond 1 MB in memory -- this particular card is using 0xfa000000 for the start of the buffer), which means everything is running through some extra assembly steps to populate the on-screen content.  It makes it pretty slow right now (about 1 second to write content to the entire screen) using my 2.2 GHz AMD64-based CPU.  Once I get the graphics engine working decently, I can switch to my more modern video card and use higher resolutions at faster speeds on its faster bus (the 3dfx card is just PCI).

ES/1 should support general purpose graphics, including a desktop, mouse pointer, flashing caret (cursor where you type), multiple text-based console windows, a mini-Presentation Manager for real graphical apps, and be able to run several separate programs simultaneously.

I'll be using my own design "dynamic scalable font" (DSF) engine for all fonts.  I plan to also have a setup tool to allow you to create your own fonts using its design (James 4:15 -- Lord willing).  My first fonts are libsf.dsf and libsf_mono.dsf, which are similar to Ubuntu's font design, but a little less bold, narrower, and with some character changes.

I'm fairly happy with how things are progressing.  It's slow and steady work.  I hope to have an ES/1 kernel booting with a pre-alpha static image in Jan.2018.  It would be developed enough to allow different people to see if it boots, has a mouse pointer, does graphics, responds to mouse, keyboard, and timer events, etc, so I could get some feedback.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 04, 2017, 02:23:08 pm
If any of you want to follow along with my development and test out what I'm doing on your local machine, do these steps:

Code: [Select]
(1)  Install IBM PC DOS 7 (or other DOS system) on one of your partitions
(2)  Install Open Watcom for DOS
(3)  Clone my libsf repository to your local C:\ drive in a Windows machine.
     Note:  It has more than just ES/1 and ES/2 in it, and it uses long filenames
(4)  Copy the \libsf\es2\ES1\ and \libsf\es2\ES2\ to your DOS partition's C:\ES1\ and C:\ES2\ folders.
     Note:  I always use a "g.bat" file and a "d.bat" in the current director for "go compile" and "debug".
     Note:  You can type "g" to begin compiling, and "d" to begin debugging.
     Note:  You'll need to setup your environment to C:\WATCOM\ or wherever you install Open Watcom.
     Note:  I will post my SETENV.BAT content when I get back on that machine.

If you don't have a copy of IBM PC DOS 7, it can still be found on places like eBay for sale.  IBM PC DOS is also available for download online (https://winworldpc.com/product/pc-dos/7x).  MS-DOS will probably work as well as other versions of DOS.  It doesn't use anything fancy, though some versions of the Watcom compiler use DPMI and DOS4GW.

Download Open Watcom for DOS (http://www.openwatcom.org/download.php).  Note:  You'll want the "open-watcom-c-dos-1.9.exe" file for download, and you'll probably want to rename it "winstall.exe" for use in DOS.  Download it in Windows and copy to your DOS partition so when you reboot it's there.

The git repository to clone is here (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse):
Code: [Select]
C:\> git clone http://www.libsf.org:8990/scm/lib/libsf.git
Cloning into libsf...

The folders to copy are here (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2).  On your local hard-drive they will appear here:
Code: [Select]
C:\libsf\> CD ES2
C:\libsf\ES2\> DIR /A:D
 Volume in drive C is LABEL
 Volume Serial Number is 1234-5678

 Directory of c:\libsf\es2

11/19/2017  11:17 PM    <DIR>          .
11/19/2017  11:17 PM    <DIR>          ..
12/03/2017  12:57 AM    <DIR>          ES1
11/12/2017  06:51 PM    <DIR>          ES2
               0 File(s)              0 bytes
               4 Dir(s)   1,310,162,944 bytes free

You'll need to use Windows Explorer or execute these commands to copy it over (assuming D: is your DOS partition, but replace D: with whatever it is):
Code: [Select]
X:\> C:
C:\> CD \LIBSF\ES2\
C:\libsf\es2\> XCOPY /S ES*\*.* D:\*.*

In the end you'll want to have a DOS partition which has at least these folders:
Code: [Select]
C:\> DIR /A:D
 Volume in drive C is LABEL
 Volume Serial Number is 1234-5678

 Directory of c:\

11/19/2017  11:17 PM    <DIR>          .
11/19/2017  11:17 PM    <DIR>          ..
11/12/2017  06:51 PM    <DIR>          DOS
12/03/2017  12:57 AM    <DIR>          ES1     <=== ES/1 development
11/12/2017  06:51 PM    <DIR>          ES2     <=== ES/2 development
11/12/2017  06:51 PM    <DIR>          WATCOM
               0 File(s)              0 bytes
               6 Dir(s)   1,310,162,944 bytes free

Once there you can navigate to "\es1\disk\" and type "g.bat".  It probably won't work properly on your machine because it's hard-coded right now to the 3dfx voodoo3 2000 and VESA mode 0x11b is my 1280x1024x24 mode.  But, I will soon be removing the hard-codings and you can use the software.

In the mean-time you can examine the source code.  The main files are:
Code: [Select]
\es1\vbe\*.*
\es1\vio\*.*
\es1\common\*.*
\es1\disk\*.*
\es1\keyflags\*.*

I'm using The SemWare Editor Jr. for DOS (http://www.semware.com/html/tsejrfl.html) to do my editing, though I actually have an old version from the 1990s before the author made The SemWare Editor for Windows, so mine is actually just called The SemWare Editor.  It was written by a man name Sammy Edward Mitchell ("SEM" in SemWare), from Marietta, Georgia back in the late 80s.  It used to be sold under the name "QEdit" in the BBS days.

-----
If any of you want to start up a dialogue with me and ask questions about development or how to do something, you can reach me at my email address.  It is rick.c.hodgin and I use GMail.  If you can put those together you'll be able to reach me. :-)
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 05, 2017, 02:50:09 pm
I plan on creating a real-mode emulator that will run in 32-bit protected mode, allowing for all existing 16-bit code to run natively in ES/2 through emulation.  Drivers can be supported, but not the full BIOS machine state.  We'll address each limiting factor one-by-one as they come up, and will support everything necessary to make all existing OS/2 drivers work, but it won't be a full machine emulation.  Just enough to make everything that currently exists work properly.

I will begin development on the 16-bit emulator as a later part of ES/1 development.  It will be called Em86+, and will serve as one of the assets available to ES/2 in moving forward.

I also plan a full flat 32-bit 386-like ISA emulator (an 80386 environment with some ISA extensions, called Em386+) which will allow code to be written for a virtual 386+ machine that can be suspended, and resumed at any state.  It will not run at native machine speed but in emulation at probably 30-50% machine speed, but it would be able to run as required, in a known environment, and on every type of hardware natively.

Em386+ will be a much simpler and somewhat extended 80386 environment, with additional instructions, 3x the number of registers, and a flat address space beginning at offset 0 and continuing up.  The ES/1 and ES/2 compilers will be able to write code which targets this environment.

One advantage of using Em86+ and Em386+ will be when there is an error, the machine state can be saved at that very point.  You can also enable debug traces to diagnose the last several thousand instructions to figure out what sequence of events allowed something to arrive at an error state.

I eventually plan to incorporate these new 86+ and 386+ environments into my Arxoda CPU design, allowing them to eventually run on native hardware at full machine speed.  They will also include debug trace abilities.  Long term goals.
Title: Re: ES/1 open source kernel
Post by: ak120 on December 05, 2017, 06:00:36 pm
I plan on creating a real-mode emulator that will run in 32-bit protected mode, allowing for all existing 16-bit code to run natively in ES/2 through emulation.
Sorry, but it sounds quite strange. Some wrong assumptions as usual. You claim that "all existing 16-bit code" should "run natively" by a fictive real-mode emulation. But on the other hand not "all existing 16-bit code" operates in real-mode and additionally you can write real-mode code that can deal with 32-bit registers. Every 386 compatible CPU has a virtual 8086 mode which also includes x86-64.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 05, 2017, 06:05:37 pm
I plan on creating a real-mode emulator that will run in 32-bit protected mode, allowing for all existing 16-bit code to run natively in ES/2 through emulation.
Sorry, but it sounds quite strange. Some wrong assumptions as usual.

Perhaps on your part as well, Andreas Kohl.

Quote
You claim that "all existing 16-bit code" should "run natively" by a fictive real-mode emulation. But on the other hand not "all existing 16-bit code" operates in real-mode and additionally you can write real-mode code that can deal with 32-bit registers. Every 386 compatible CPU has a virtual 8086 mode which also includes x86-64.

I am well aware of that.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 06, 2017, 04:56:10 am
I have been able to get the graphics working in 24-bit and 32-bit graphics modes.  The 3dfx card is in a PCI bus and takes about one second to redraw the screen in 1280x1024x24 mode.  The ATI card uses an AGP bus and updates a 1600x1200x32 screen in about 1/10th second in full-redraw mode.

I'll be moving entirely to a graphics mode for all future development.  Am working on adding mouse and flashing caret support.

ES/1 will support all apps, both text-based and graphics, as windows on the desktop.

UPDATE:  I have been able to get 16-bit working too.  I've also added a delta render function, allowing console windows to work in graphics mode by only rendering changed characters or attributes.  With delta render, on console windows it's as fast as text now.  Working on mouse pointer and flashing caret now.
Title: Re: ES/1 open source kernel
Post by: Dave Yeo on December 06, 2017, 08:23:34 am
What object format are you using? Also what executable format?
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 06, 2017, 08:40:14 am
What object format are you using? Also what executable format?

Right now I'm doing all work in DOS with the Open Watcom for DOS compiler.  It uses OMF and ELF by default.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 06, 2017, 02:46:46 pm
I've contacted Sammy Mitchell who wrote The SemWare Editor, Jr. for DOS.  He's going to make me a special version that receives -rows and -cols command line parameters and uses those rather than BIOS and INT 10h services to determine the screen size.

He's going to modify TSE, Jr. for DOS to allow those parameters and write to text video memory at 0xb800000.  I will write a TSR which projects that memory into graphics mode, allowing a 200x75 console and editor.

Since I have my delta render working, it should be as fast as text in DOS.

Here is the source code (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/ES1/VBE/VBE.H).  It's missing some recent changes, but those will be pushed tonight.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 09, 2017, 11:41:39 pm
I have the mouse pointer working correctly, on text and graphics modes.  Am developing the flashing caret algorithms now.

Once completed, I'll finish development on the Vio algorithms for the menu and input fields.

This will give us a solid framework for text-based UI in a text or GUI wrapper.  I will use that base for developing all ES/1 apps.  The first will be kernel code support for FAT12/16 disks, and memory allocation algorithms.  The second will be my Debi debugger.  The third will be cmd.exe and supporting algorithms (copy, move, md, rd, etc.).

Here's to great success (James 4:15 -- Lord willing) !

Note:  I will say that programming back in text-based consoles in DOS these many days really makes me appreciate GUI interfaces and TrueType fonts. :-). I may bump my DSF implementation up a few notches and give myself a GUI with proportional fonts.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 10, 2017, 05:26:17 am
Here's a screenshot.  I'm using VESA video modes.  This one is 1600 x 1200 with 32-bit color.  You can see the mouse pointer rendered, and I caught the caret/cursor mid-blink so you can see it there near the mouse pointer.

ES/1 allows three types of built-in carets:  (1) boxed outline, (2) line [shown in the image], and (3) solid.  There is also a 4th type, which is a custom caret up to 16 x 32 in size.

The mouse pointers supported by default are:  (1) arrow, (2) inverse arrow [white with black outline], (3) cross, (4) I-Beam, and a custom type of up to 16 x 32 in size.

Carets and mouse pointers are all up to 254 shades of gray, so they can be smoother than what I have shown there.  You can see their default appearances here (http://libsf.org:8990/projects/LIB/repos/libsf/browse/es2/ES1/VBE/GLOBALS.H).

UPDATE:  The information at the bottom is debugging info:  mouse position, timer tick, caret position.  The caret flashes on and off and is only displayed on ticks, not tocks.  Here it's on a tick cycle.
Title: Re: ES/1 open source kernel
Post by: ak120 on December 10, 2017, 08:39:15 pm
Here's a screenshot.  I'm using VESA video modes.  This one is 1600 x 1200 with 32-bit color.

It's unbelievable. Unfortunately a colour in PNG files can only have 1, 2, 4 or 8 bits (or additionally 16 bits for grayscale). Simply run a test to count the number of colours in the attached image file.

Those who know nothing must believe everything.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 10, 2017, 10:30:26 pm
Here's a screenshot.  I'm using VESA video modes.  This one is 1600 x 1200 with 32-bit color.

It's unbelievable. Unfortunately a colour in PNG files can only have 1, 2, 4 or 8 bits (or additionally 16 bits for grayscale). Simply run a test to count the number of colours in the attached image file.

32-bits is the color depth.  It uses R,G,B,X each 8-bits.

32-bits uses 8:8:8:8 bits for R,G,B,X.
24-bits uses 8:8:8 bits for R,G,B.
16-bits uses 5:6:5 bits for R,G,B.

The image you see is a text-based console window projected into graphics modes.  It only uses 16-colors.

During development of my mouse pointer algorithms, I had some bugs at various times, and every color of the rainbow appeared. :-)

You can see the VESA algorithms I created here (http://www.libsf.org:8990/projects/LIB/repos/libsf/browse/es2/ES1/VBE/VBE.H).  Search for "bpp" to find references to various drawing algorithms.

I can create an app that renders something in colors.  Tell me what to write and I'll post the source code and instructions for how to compile it so you can run it live and see it rendering colors.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 11, 2017, 02:11:18 am
I can create an app that renders something in colors.  Tell me what to write and I'll post the source code and instructions for how to compile it so you can run it live and see it rendering colors.

I know what I'll write.  The ES/1 logo and splash screen for booting.  Here is a sample from OS/2 (https://guidebookgallery.org/pics/gui/startupshutdown/splash/os2warp3.png).

UPDATE:  Here's the initial design (http://www.libsf.org/es2/splash_es1.png).
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 11, 2017, 05:42:28 am
...Unfortunately a colour in PNG files can only have 1, 2, 4 or 8 bits (or additionally 16 bits for grayscale).

The bit depth you indicate there are per color channel bit counts (such as 1, 2, 4 or 8 bits for the red channel, and another 1, 2, 4 or 8 bits for the green, and another set for the blue, resulting in 3-bit, 6-bit, 12-bit, or 24-bit colors by your own statement).

PNG also supports extended true color RGB images with a full 16 bits per color channel, resulting in a total encoding of 48-bit samples per pixel.  And there is one which includes an alpha channel RGBA with 32- or 64-bits per pixel.

www.libpng.org -- Chapter 8 -- PNG Basics -- 8.5.6 RGB (http://www.libpng.org/pub/png/book/chapter08.html#png.ch08.div.5.6)
Title: Re: ES/1 open source kernel
Post by: ak120 on December 13, 2017, 01:11:01 am
32-bits is the color depth.  It uses R,G,B,X each 8-bits.
But only R, G and B are colours - it's still 24bits of colour depth summarised.

By using 10 bits you would get 30bits totally. But 32bits is very uncommon - I cannot think of any practical example.

Quote
The image you see is a text-based console window projected into graphics modes.  It only uses 16-colors.
Hard-coded or per virtual screen?

I know of Greg's book but prefer the paper form. ;-)
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 13, 2017, 10:42:10 am
32-bits is the color depth.  It uses R,G,B,X each 8-bits.
But only R, G and B are colours - it's still 24bits of colour depth summarised.

By using 10 bits you would get 30bits totally. But 32bits is very uncommon - I cannot think of any practical example.

32-bits is used everywhere.  It is far faster and simpler to code, and automatically maintains alignment in memory.  Some ATI VESA implementations only support 16-bit and 32-bit for data alignment.

Quote
The image you see is a text-based console window projected into graphics modes.  It only uses 16-colors.
Hard-coded or per virtual screen?

Hard-coded presently to emulate VGA BIOS color defaults, but easily extensible.

Since your comment about colors, I have had it in mind to provide calls which allow arbitrary rgba() colors as well.  They would work on graphics consoles by overriding the default mapped color attributes.  I am also adding support for disabled, and underlined text in graphics mode.  Disabled will dither.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 15, 2017, 10:18:17 pm
For ES/1, I have developed a fairly comprehensive graphics engine that runs entirely under VESA.  I'm currently working on a fonts engine to support more than 8x8, 8x14, and 8x16 fixed-point console fonts.  I plan to have a proportional "libsf" font, and a fixed point "libsf mono" font at release, in addition to the three console fonts.

As such, ES/1 will now launch with a start bar, a few built-in programs, and the ability to run multiple graphical command windows simultaneously.  I will begin working on a primitive Presentation Manager interface, which will allow non-console windows which contain graphics and special drawing abilities.

I plan for ES/1 to be a very simple 32-bit kernel which initially allows 16-bit apps to run using Dos*() and other function calls to simulate BIOS, but over time is expanded into a full 32-bit OS as indicated below.

Code: [Select]
The plans are:
Version 0.1 -- boot and run from a floppy disk
Version 0.2 -- boot from a floppy or FAT16 hard disk
Version 0.3 -- adds additional apps to support the kernel
Version 0.4 -- introduction of a general purpose text editor, compiler, and assembler for creating custom ES/1 apps
Version 0.5 -- increased development of the Presentation Manager
Version 0.6 -- introduction of system objects
Version 0.7 -- a fully integrated desktop and the introduction of 32-bit apps
Version 0.8 -- network stack, sockets, ES/1 network file system support
Version 0.9 -- boot from a network drive, some usable userland apps
Version 1.0 -- a functional general purpose OS release

I plan for each ES/1 release will be available as a downloadable virtual machine image with virtual media already configured, able to run under Oracle's VirtualBox.  Or, as stand-alone boot floppy images for real machine installation.  I plan for Version 1.0 to be able to run multiple apps well, possess a robust desktop user experience within its fully object-oriented design, giving it a general look and feel similar to that of OS/2's desktop.

ES/1 will serve as the base for the Al-x kernel extended development, and eventually as the base for what will become ES/2 as those innate abilities are refactored for the needs of a backward binary-compatible OS/2 kernel replacement, and later driver replacement and user apps replacement.

The goal for ES/1 is to be a launching platform for the rest of development, which will eventually yield a complete and fully-replaced OS/2 kernel and suite of drivers, utilities, apps, everything.  I look forward to having other developers come on board as development progresses.
Title: Re: ES/1 open source kernel
Post by: Alfredo Fernández Díaz on December 16, 2017, 01:23:58 pm
Just a quick question: unicode?
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 16, 2017, 01:33:37 pm
Just a quick question: unicode?

For ES/1, partially supported through limited code page support through libsf and libsf-mono fonts.  I am creating a new font editor tool.  The more help I get in authoring characters for other languages, the more support there will be.  I wil unveil the font tool next.  You will see limited ES/1 updates until then.

For ES/2:  Planned to be fully supported.  I like Microsoft's ability to have funcA() for ascii, and funcW() for unicode.  There will be full kernel support for unicode, including filenames and all tag names.  Everything that can be asciiz can also be unicode, except conversion functions. :-)
Title: Re: ES/1 open source kernel
Post by: Alfredo Fernández Díaz on December 16, 2017, 01:58:16 pm
For ES/1, partially supported through limited code page support through libsf and libsf-mono fonts.  I am creating a new font editor tool.  The more help I get in authoring characters for other languages, the more support there will be.  I will unveil the font tool next.  You will see limited ES/1 updates until then.
You may want to check Alex Taylor's bitmap and vector fonts -- I think his license would let you relieve quite some work from that front. Custom tools or not, I don't think you have developed a third way to define glyphs* so you probably can re-use work.

*You haven't, have you? :)

Quote
For ES/2:  Planned to be fully supported.  I like Microsoft's ability to have funcA() for ascii, and funcW() for unicode.  There will be full kernel support for unicode, including filenames and all tag names.  Everything that can be asciiz can also be unicode, except conversion functions. :-)
Just like it is a subset of all codepages, ASCII is also a subset of unicode (ASCII chars are unicode chars 0-127), so why do you need funcA?
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 16, 2017, 02:22:42 pm
For ES/1, partially supported through limited code page support through libsf and libsf-mono fonts.  I am creating a new font editor tool.  The more help I get in authoring characters for other languages, the more support there will be.  I will unveil the font tool next.  You will see limited ES/1 updates until then.
You may want to check Alex Taylor's bitmap and vector fonts -- I think his license would let you relieve quite some work from that front. Custom tools or not, I don't think you have developed a third way to define glyphs* so you probably can re-use work.

*You haven't, have you? :)

I began with fundamentals.  I designed everything up from there.  I did what made sense to me.  Others can judge for themselves the nature of its originality or usefulness.

Quote
For ES/2:  Planned to be fully supported.  I like Microsoft's ability to have funcA() for ascii, and funcW() for unicode.  There will be full kernel support for unicode, including filenames and all tag names.  Everything that can be asciiz can also be unicode, except conversion functions. :-)
Just like it is a subset of all codepages, ASCII is also a subset of unicode (ASCII chars are unicode chars 0-127), so why do you need funcA?

ASCII goes 0..255 in my opinion. :-)  It never made sense to hobble existing apps which use those characters so they'd have to be significantly rewritten in moving forward.  Less of an issue now, but still ... it's a sound goal.

I'm not trying to piece together what already exists because it's faster.  I'm trying to build up a product from the ground, founded on the Lord, given over to this project for the people, so that all of it is built from within those who will contribute, for that purpose, on purpose.  In that way its origins are solidly given over in that way, with a common thread of purpose throughout.
Title: Re: ES/1 open source kernel
Post by: Alfredo Fernández Díaz on December 16, 2017, 02:55:42 pm
ASCII goes 0..255 in my opinion. :-)
But that's what the codepage mess was about from the start. All codepages I know of* (classic ones, I don't know if the term has caught on a new meaning lately) are 255-character sets, and every character on any of them is therefore represented by one byte. And that's the problem: simplifying quite a bit, you only 'know' character 0x65 is an 'A' because ALL codepages have 'character 0x65 = A' in their definition. But the answer to 'what is character #128' (or whatever > 2^7 -1 = 127)? depends on what codepage you are using. You can't just put say Spanish and Russian ASCII text together like that because the same numbers above 127 represent different characters, etc. Unicode was conceived to get rid of those problems, but had to be retro-fitted unto existing systems, which caused problems of its own.

If you asked me, my own Unicode implementation would be very different from what we have currently, but that's another story. If you are beginning something from scratch, and want it to be able to eventually 'talk' to the rest of existing software, proper Unicode support is a must because it is nearly universal now.

*I am leaving DBCS CPs aside for the sake of simplicity. These are of course 65535-char codepages, hence 'DBCS'.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on December 16, 2017, 03:00:41 pm
If you asked me, my own Unicode implementation would be very different from what we have currently, but that's another story. If you are beginning something from scratch, and want it to be able to eventually 'talk' to the rest of existing software, proper Unicode support is a must because it is nearly universal now.

What would you create?  OS/2 supported unicode, albeit clunkily.  It would be a reversal of direction to replace it.  It's also completely prevalent today.  If any apps would be ported, support would have to exist.
Title: Re: ES/1 open source kernel
Post by: Alfredo Fernández Díaz on December 16, 2017, 03:13:34 pm
Sorry, we can discuss my own ideas of how we could have a better Unicode some other day, it would require much thinking and long-term plans of interoperability -- assuming it would actually be better ;)

What I meant to say is that any new thing of yours that wants to talk to other software today must readily implement Unicode as we have it today (or interfaces to it), and not get stuck in the codepages stage. If it is not, fine :)
Title: Re: ES/1 open source kernel
Post by: ak120 on December 16, 2017, 04:28:54 pm
ASCII goes 0..255 in my opinion. :-)
7-bit ASCII was always from 0 to 128. But ISO-8859-1 (CP819) offers 8-bit.

Quote
But that's what the codepage mess was about from the start. All codepages I know of* (classic ones, I don't know if the term has caught on a new meaning lately) are 255-character sets, and every character on any of them is therefore represented by one byte.
That's the special case for 8-bit SBCS codepages - it's a good idea to distinguish between codepage and char set.

Quote
And that's the problem: simplifying quite a bit, you only 'know' character 0x65 is an 'A' because ALL codepages have 'character 0x65 = A' in their definition.
I can not follow at this point. Only a subset ASCII derived codepages behave this way. Decimal 65 for 'A' fine but in EBCDIC 'A' is 193.

Quote
But the answer to 'what is character #128' (or whatever > 2^7 -1 = 127)? depends on what codepage you are using. You can't just put say Spanish and Russian ASCII text together like that because the same numbers above 127 represent different characters, etc.
ASCII is a stupid American thing. For Russian there is GOST or better use Unicode. If you want to manage it with 7-bit ASCII you can do it with TeX.

Quote
*I am leaving DBCS CPs aside for the sake of simplicity. These are of course 65535-char codepages, hence 'DBCS'.
But nowadays MBCS are quite common or call it variable character size encodings.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on September 18, 2018, 11:09:32 pm
Just a quick update.

This project has not gone away and will continue.  I am currently working on developing a general purpose compiler called CAlive that supports (C/C++)-like code.  It will serve as the base compiler used for x86, as well as my own VM I'm working on (called Inspire VM), and will serve as the compiler I use to compile the user OS/2 source code people have for creating binaries that run in ES/2.

ES/1 will serve as a simple test platform for developing and testing new designs that will filter into ES/2.

If any of you would like to help, my plans are to continue development of ES/1 and ES/2 after I get my compiler completed, which is targeted for sometime on or around Christmas 2019.

Thank you. :-)
Title: Re: ES/1 open source kernel
Post by: David Graser on October 06, 2018, 07:25:15 pm
Rick

Hopefully your new kernel will eventually do away with the SIQ. 
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 26, 2018, 09:18:32 pm
Quote from: Rick C. Hodgin
Nov.13.2017I've begun coding a DOS-like ES/1 operating system as a precursor to my ES/2 kernel.  It will be a simple text-based kernel in real-mode allowing me to port compilers, assemblers, linkers, debuggers, to that system, allowing me to test code in a simple environment, one I can easily mimic in IBM PC DOS 7.

ES/1 will also be released as a simple public OS people will be able to download and use for FAT12 and FAT16 disks.  It will also natively allow serial port networks. :-)

I've revisited this plan and have been working on the ES/1 kernel again.  I'm working in IBM PC DOS 7 with Open Watcom 1.9 on the graphics engine.  ES/1 will support basic graphics (windows, controls, be event-driven), as well as console windows displayed in a graphics window.

I'm currently working on my FAT12/16 driver to allow hard disk access without BIOS support.  Once that is completed, I'll begin working on supporting many functions present in the DOS INT 21h API, and will present a system people can download and try.

My goal is to use ES/1 as a platform for developing base ES/2 algorithms, but real development on ES/2 won't begin until late 2019 or so.  I'd like to have ES/1 booting, accessing hard disks, using VESA to support video modes, and handling all forms management in software, and get it working on as many types of machines as possible (those with traditional BIOS).

Once it's up and running, I'll begin creating tools which work in the ES/1 environment, and if there are others who would like to port other tools then I can extend the API to support whatever they need.

ES/1 intends to be a fully pre-emptive 32-bit multi-tasking OS running on a single core initially, with SMP added in mid-/late-2019.  It will support 16-bit and 32-bit code, including real mode code in v86.  I may never make a 64-bit ES/1 version, but ES/2 will be a 32-bit / 64-bit hybrid out of the box.  The user will actually be able to choose to boot into either the 32-bit or 64-bit kernel with a startup option.
Title: Re: ES/1 open source kernel
Post by: RickCHodgin on November 27, 2018, 12:03:40 pm
In development of the ES/1 operating environment, I've run into a limit in Open Watcom being able to compile the program in DOS as monolithic.  Am going to have to break it out into separate modules.