Author Topic: HPFS and JFS internal definition specifications  (Read 1739 times)

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #15 on: June 02, 2017, 03:13:39 pm »
There already exists a "large file" API and there also exists additional entry points and extensions for existing entry points for file system drivers to support these new APIs.
These APIs will use 64-bit signed values to deal with file size and seek offsets.

The only problem is that the current HPFS.IFS was written at a point in time where these additional entry points where not yet defined.
If these additional entry Points were added to the existing HPFS.IFS then it would not be necessary to use the "DEADBEEF hack" and read in units of sectors instead of bytes.

As an example, there is an effort currently going on to add these additional "large file" entry points to the existing FAT32.IFS.
That will finally allow to deal with files up to 2^63 Bytes and likewise will allow byte granular accesses/seeks across disks up to a size of 2^63 bytes.

Well, since I'm essentially considering creating a new open-source HPFS from scratch with full backward compatibility, it would not be hard to add those newer extensions, and to allow for larger files.

We'll see.  I'm still waiting to hear back from Martin and the general community on what they think I should pursue.  I'm interested in both the USB and HPFS so far.  I want to take a look at JFS and ZFS as well.

Thank you,
Rick C. Hodgin

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 38
  • Posts: 928
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #16 on: June 02, 2017, 04:23:41 pm »
Quote
I'm still waiting to hear back from Martin and the general community on what they think I should pursue.

It may be more useful to discuss the situation with Arca Noae. They are the ones who are driving the platform forward. No point in duplicating what they are doing, and they can use the assistance of a programmer who knows how to do such things.

HPFS and JFS are mature, and working, file systems. FAT32 is never going to work properly, because MS owns it, and won't tell how it actually works. There has been discussion of porting EXT-x file systems (and some others). Whether that would actually be useful, or not, is debatable. Other things are far more important.

IMO, at the moment, USB 3.x might be the best place to go, but Lars Erdmann  is doing most of the USB development, so you may want to talk to him about the possibilities. Arca Noae is doing the "stable" release, mostly based on Lars' work.

The other major need is for WiFi, but I understand that Arca Noae has got that well under way.

Another major project, is to support GUID Partition Table (GPT) disks. I know that Arca Noae has looked at it, but I don't know if any development has been done.

Tom

  • Newbie
  • *
  • Thank You
  • -Given: 0
  • -Receive: 2
  • Posts: 13
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #17 on: June 02, 2017, 04:30:01 pm »
Does anybody know if Gordon Letwin is still alive?  He wrote the HPFS specification and is from Indianapolis, IN where I'm from! :-)

He is born in july 1952, so probably still alive (he was in 2011).

Information about his original HPFS can be found in the patent:

www.google.com/patents/EP0416445A2?cl=en

In later years (Warp 3ish) Doug Azzarito has also worked on HPFS.

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 7
  • -Receive: 87
  • Posts: 1098
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #18 on: June 02, 2017, 04:32:30 pm »
You might want to look at http://hobbes.nmsu.edu/download/pub/os2/system/drivers/filesys/ext2_240.zip and the update http://hobbes.nmsu.edu/download/pub/os2/system/drivers/filesys/32drv170.zip.
At one point this ext2 driver worked well, could even boot OS/2 from it with a case non-sensitive hack and no WPS (no EA support back then). It was written to be a general driver, unluckily pre-LVM so it broke with the update to LVM.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #19 on: June 02, 2017, 04:37:33 pm »
Quote from: Rick C. Hodgin
I'm still waiting to hear back from Martin and the general community on what they think I should pursue.

It may be more useful to discuss the situation with Arca Noae. They are the ones who are driving the platform forward. No point in duplicating what they are doing, and they can use the assistance of a programmer who knows how to do such things.

I contacted Arca Noae about this time last year when I was moving forward toward writing my open source OS/2 kernel replacement.  They were very non-receptive to the idea and it was a difficult email exchange over a week or two.  If they're willing to guide me in some direction, I'd consider it.  I love the new Arca OS release.  And the idea of having a modern OS/2 is very appealing to me.  It's the most exciting idea I have in anything related to operating systems and general day-in/day-out computer use.

Thank you,
Rick C. Hodgin

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 38
  • Posts: 928
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #20 on: June 02, 2017, 05:58:55 pm »
Quote
I contacted Arca Noae about this time last year when I was moving forward toward writing my open source OS/2 kernel replacement.  They were very non-receptive to the idea and it was a difficult email exchange over a week or two.

Well, first off, the kernel is not likely to be a short term project, and AN doesn't have the manpower to manage such a project over the longer term. If you want to do that, and present them with a working kernel, I expect that they would look at it. The main problem being that it would not be certified to the required standards, demanded by many of their customers. I doubt if there are enough OS/2 users left to be able to pull that off. It is also much more important to add things like USB 3.x and WiFi, in the short term, so that people can actually use the product. AN do need to get the basic OS running well enough to keep their customers happy, and that must be done in the short term, with very limited manpower. It must also be done with no possibility that somebody else will claim that their work was copied, or, they need a complete disclaimer that all work is original, and they have full access to whatever is presented to them. They do have contracts with IBM, other software vendors, and customers, that need to be followed. They cannot take any chances on something like a new kernel, that may, or may not, work as expected, in all cases (and they do understand just how difficult that project would be - I wonder if you understand that). It is difficult enough to get enough testing done on things like NIC drivers, to keep customers happy. Writing software is only the beginning of what needs to be done to add parts to a commercial product that may be used in military, and health care, situations, where no failures can be tolerated. Small steps are necessary, to keep OS/2 alive, in the short term, and there aren't enough people to do long term projects, never mind get them certified.

To write a new kernel, you need to start with a description of ALL of the things that you expect it to do. I honestly doubt if anybody on earth could describe all of the things that the OS/2 kernel does. If the new kernel is not 100% compatible, it really cannot be called "OS/2" (in fact, AN uses the name ArcaOS because IBM forbids using "OS/2"). A new kernel that can run OS/2 programs would probably be accepted by a lot of users, but the main, core, customers would not be able to use it until it is properly certified.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #21 on: June 02, 2017, 06:07:05 pm »
Quote from: Rick C. Hodgin
I contacted Arca Noae about this time last year when I was moving forward toward writing my open source OS/2 kernel replacement.  They were very non-receptive to the idea and it was a difficult email exchange over a week or two.

Well, first off, the kernel is not likely to be a short term project, and AN doesn't have the manpower to manage such a project over the longer term...

To be clear, I wasn't asking them to be involved with the project in any way, but was asking for some information, general help, guidance as to where to begin, etc., basically just as fellow members of the community.

I also never intended for it to be binary compatible.  It would become the root of a new open source OS/2 derivative that I have been calling ES/2.  It would be designed around every feature of OS/2 at its core (at least those that are publicly available), to produce an extremely similar environment to that of OS/2, but it would be its own off-shoot, kind of like how Linux is like UNIX, but it is not UNIX and there are distinct differences.

I still want to do that project.  I was telling my wife last night that if I had resources to do all the things I want to do, I would manage all of them, but the one thing I would personally write the bulk of the code for would be the kernel.  The rest of them I would write guidance and frameworks, but not details of the code.

Thank you,
Rick C. Hodgin

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #22 on: June 02, 2017, 06:27:41 pm »
I restored the original fig5 and fig6 from the original source on the HPFS article. I don't what if the optimized images got list or maybe never was finished. I will check on that.
...I'll go ahead and create some new optimized images and you can update them when completed.

Emailed to you.  Taken from the link:  Google Patent #EP0416445A2

Thank you,
Rick C. Hodgin

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Thank You
  • -Given: 244
  • -Receive: 46
  • Posts: 1642
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: HPFS and JFS internal definition specifications
« Reply #23 on: June 02, 2017, 06:50:11 pm »
Hi Rick.

My suggestions about file systems are the one that I posted.  If I will have to put those (HPFS, Boot JFS and ZFS) on priority order, it looks more interesting ZFS to me, since HPFS has MS patents on it and JFS is close source but AN is supporting it. ZFS can be focused on being an alternative for the future OS/2 file system. 

But while there are people saying "don't reinvent the wheel" I say "reinvent the wheel if it is not open source". Close source software is future abandonware to me, even if it works fine today. (Already discussed on the past how open source software reduce the risk to be left "high and dry")

Gordon Letwin, according to wikipedia, is a guy that made millions with Microsoft, so I found it a little bit hard to contact him to request HPFS source code, his OS/2 book, or any other things related to OS/2 to be turn open source/public domain. He seems to be more focused on environmental activism. But I had never tried to contact him.  Interesting historic post by Gordon Letwin, by the way.

If I can choose a priority outside the "File Systems" the priority should be to start cloning more deep components of OS/2 like the CPI API  (possible more based on libc library) and a PM/GPI clone (also with more close dependencies to libc instead of CPI). It will require developers with a good understanding of the CPI API, PM API and LIBC. But since I don't do development and do not have money for an army of developers, this is only "sweet sweet drunk talk" for the moment.

Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #24 on: June 02, 2017, 07:10:28 pm »
Hi Rick.

My suggestions about file systems are the one that I posted.  If I will have to put those (HPFS, Boot JFS and ZFS) on priority order, it looks more interesting ZFS to me, since HPFS has MS patents on it and JFS is close source but AN is supporting it. ZFS can be focused on being an alternative for the future OS/2 file system. 

But while there are people saying "don't reinvent the wheel" I say "reinvent the wheel if it is not open source". Close source software is future abandonware to me, even if it works fine today. (Already discussed on the past how open source software reduce the risk to be left "high and dry")

Gordon Letwin, according to wikipedia, is a guy that made millions with Microsoft, so I found it a little bit hard to contact him to request HPFS source code, his OS/2 book, or any other things related to OS/2 to be turn open source/public domain. He seems to be more focused on environmental activism. But I had never tried to contact him.  Interesting historic post by Gordon Letwin, by the way.

If I can choose a priority outside the "File Systems" the priority should be to start cloning more deep components of OS/2 like the CPI API  (possible more based on libc library) and a PM/GPI clone (also with more close dependencies to libc instead of CPI). It will require developers with a good understanding of the CPI API, PM API and LIBC. But since I don't do development and do not have money for an army of developers, this is only "sweet sweet drunk talk" for the moment.

Regards

HPFS is very intriguing to me.  I'm going to continue learning about it to see if it continue to pique my interest.  It would be awesome to have an open source, well-debugged, high performance file system that could be ported to pretty much any OS with just a few tweaks.

I went through my old IDE hard drives last night looking for my PCI utility, but of the 10 drives I have, five were bad, three worked, and I have two left to test.  I couldn't find the content on the three drives that work.

Nonetheless, after booting up in DOS on those three drives last night, I had the thought I would create a FAT16 partition and install DOS, and a second HPFS partition and install OS/2, and then have files on there and begin writing my driver to be able to read that second partition, and eventually read and write to it, and then have it working and updating files in a way that OS/2 then sees when I boot back into it.

I might be able to accomplish the same thing looking at virtual machine disk files, but I'm not sure of their data format.  If they're linear, then it would be doable.  We'll see.

Thank you,
Rick C. Hodgin

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 7
  • -Receive: 87
  • Posts: 1098
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #25 on: June 03, 2017, 02:40:12 am »
@Martin, any HPFS patents should have ran out by now. They're generally for 20 years.

@Rick, don't forget there is a GPL HPFS driver in the Linux kernel.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #26 on: June 03, 2017, 02:41:32 am »
@Rick, don't forget there is a GPL HPFS driver in the Linux kernel.

I'm going over the patent right now, converting it from scanned image pages to text.  It's only 329 pages. LOL!  At least it's double- and triple-spaced mostly. :-)

Thank you,
Rick C. Hodgin

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #27 on: June 03, 2017, 10:31:23 pm »
It looks like the best thing for me to do is create my own custom disk format for development of this driver.  I can use a type that doesn't presently exist, and then create a driver which is able to access it using my own file system.

Once that is working and OS/2 is able to recognize the information on that disk through the API, an the API is all working correctly, then I can modify the algorithms to access the data on HPFS disks.

I'm really enjoying learning about HPFS.  I've always wondered why OS/2 hard disk access patterns sounded notably different than FAT32.  Now I know. :-)

Thank you,
Rick C. Hodgin

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 7
  • -Receive: 87
  • Posts: 1098
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #28 on: June 04, 2017, 01:26:18 am »
HPFS was one of the big reasons that made OS/2 a better DOS and Windows (3.x). Much faster then FAT.
It's a shame that MS kept HPFS386 and charged a $1000 for a license.
The story was that IBM and MS decided that they would each develop a new file system, written in C and capable of running on a 286 and which ever one was faster would be used. MS wrote HPFS386, won the contest by cheating as usual and IBM had to rewrite the HPFS driver and gave it the small 2MiB cache.
Hopefully you put some thought into the cache, could even use memory above 4GB I guess.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #29 on: June 04, 2017, 01:47:00 am »
HPFS was one of the big reasons that made OS/2 a better DOS and Windows (3.x). Much faster then FAT.
It's a shame that MS kept HPFS386 and charged a $1000 for a license.
The story was that IBM and MS decided that they would each develop a new file system, written in C and capable of running on a 286 and which ever one was faster would be used. MS wrote HPFS386, won the contest by cheating as usual and IBM had to rewrite the HPFS driver and gave it the small 2MiB cache.
Hopefully you put some thought into the cache, could even use memory above 4GB I guess.

I promise you, I'm going to do it right.  And, it will be flexible and extensible.  I might even like to create a merged HPFS / JFS driver that handles both disk types.  But, we'll see.

As I start working on this I'm back to wanting to replace the kernel first.  There are significant limitations in trying to write and debug this type of code when you don't have access to the kernel source code.  It hampers you immensely.

Ugh.

Thank you,
Rick C. Hodgin