Author Topic: HPFS and JFS internal definition specifications  (Read 1667 times)

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
HPFS and JFS internal definition specifications
« on: June 01, 2017, 04:48:39 pm »
Can somebody point me to a link for HPFS and JFS specs showing the physical layout of data in those file formats?

And is there an open source implementation of these file systems?  I'd like to write one if not.

Thank you,
Rick C. Hodgin



Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 7
  • -Receive: 87
  • Posts: 1095
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #1 on: June 01, 2017, 05:52:37 pm »
There's a fairly complete HPFS implementation in the Linux kernel, and of course, the JFS code in the Linux kernel is a fork of our JFS. The fork happened with this code, https://github.com/OS2World/DRV-JFS-GPL
Note that the Linux JFS is no longer compatible with our bootable JFS.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #2 on: June 01, 2017, 06:03:44 pm »
There's a fairly complete HPFS implementation in the Linux kernel, and of course, the JFS code in the Linux kernel is a fork of our JFS. The fork happened with this code, https://github.com/OS2World/DRV-JFS-GPL
Note that the Linux JFS is no longer compatible with our bootable JFS.

How much do you know about these file systems?

Thank you,
Rick C. Hodgin

Dariusz Piatkowski

  • Sr. Member
  • ****
  • Thank You
  • -Given: 12
  • -Receive: 6
  • Posts: 268
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #3 on: June 01, 2017, 10:36:53 pm »
Rick,

What specific information are you looking for?

I spent some time writing an enhanced version of 'cache386 /s' command. Not quite sure to be honest if I ever finished this, it was really more of a learning effort to undertsand OS/2, the programming platform, etc.

Anyways...since I have been a heavy HPFS386 user I have some pretty good amount of docs on these. Most of it sourced on-line, some of it may still float out there. Bulk of this is system calls/APIs though...

guzzi

  • Sr. Member
  • ****
  • Thank You
  • -Given: 0
  • -Receive: 12
  • Posts: 260
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #4 on: June 01, 2017, 11:18:05 pm »
Jan van Wijk, author of DFSee, is likely very well acquinted with the structures of several file systems. You might drop him a line.

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Thank You
  • -Given: 243
  • -Receive: 46
  • Posts: 1637
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: HPFS and JFS internal definition specifications
« Reply #5 on: June 02, 2017, 12:24:08 am »
Hi Rick

If you have interest on developing a file system for OS/2-ArcaOS I may suggest the following ideas.

1) HPFS. Even that we have an HPFS file system driver, there is no open source implementation of it. You may get some information of HPFS also at the EDM/2.

2) Bootable JFS: Even that we already have that available, it is not open source. It is based on IBM's DDK source code which can not be open to the public and non-open source compatible. Maybe evolving the JFS open source project so it can boot can be an interesting project.

3) ZFS : I hear that ZFS is an interesting file system format and there is the OpenZFS project. Maybe ZFS can be an interesting alternative to HPFS and JFS to boot OS/2 from that.

Just some thoughts.

Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #6 on: June 02, 2017, 12:30:36 am »
Hi Rick

If you have interest on developing a file system for OS/2-ArcaOS I may suggest the following ideas.

1) HPFS. Even that we have an HPFS file system driver, there is no open source implementation of it. You may get some information of HPFS also at the EDM/2.

2) Bootable JFS: Even that we already have that available, it is not open source. It is based on IBM's DDK source code which can not be open to the public and non-open source compatible. Maybe evolving the JFS open source project so it can boot can be an interesting project.

3) ZFS : I hear that ZFS is an interesting file system format and there is the OpenZFS project. Maybe ZFS can be an interesting alternative to HPFS and JFS to boot OS/2 from that.

Just some thoughts.

Your thoughts are appreciated.

About a year ago you made some suggestions that instead of me writing my own replacement open source kernel for OS/2, that I work on some OS/2 applications and other things.  At that time it was beyond my area of interest because writing a kernel and doing other types of OS/2 programming are really very different tasks even though they might seem similar.  I still want to write the replacement kernel, but until then I would like to work on some smaller tasks that I can complete in a week or a few weeks max.

You had offered some ideas or suggestions, so let me throw the whole thing back to you and the community again and ask generally, a year later even:  What would the community like best for me to work on?

I'll start there and move forward.

Thank you,
Rick C. Hodgin

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #7 on: June 02, 2017, 12:43:53 am »
1) HPFS. Even that we have an HPFS file system driver, there is no open source implementation of it. You may get some information of HPFS also at the EDM/2.

In initially looking at HPFS it is intriguing. :-)  Documentation shows it is limited to 64 GB as implemented, but 2 TiB is theoretically possible.

It would be exciting. :-)

Thank you,
Rick C. Hodgin

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #8 on: June 02, 2017, 01:12:45 am »
Does anybody know if Gordon Letwin is still alive?  He wrote the HPFS specification and is from Indianapolis, IN where I'm from! :-)

I wonder if he's available for consultation?  Maybe in his retirement he's pining back for his old OS/2 days again and would be willing to work on helping me write my kernel and implement all manner of drivers?  I would love it.  Love it I say! :-)

Thank you,
Rick C. Hodgin

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #9 on: June 02, 2017, 01:28:53 am »
Martin, do you have these HPFS folder files somewhere?  This one is referenced specifically (www . os2world . com / images / hpfs / fig5 . png):


It's missing, and it's linked on this page:
http://www.edm2.com/index.php/HPFS

Thank you,
Rick C. Hodgin

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 7
  • -Receive: 87
  • Posts: 1095
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #10 on: June 02, 2017, 02:31:08 am »
1) HPFS. Even that we have an HPFS file system driver, there is no open source implementation of it. You may get some information of HPFS also at the EDM/2.

In initially looking at HPFS it is intriguing. :-)  Documentation shows it is limited to 64 GB as implemented, but 2 TiB is theoretically possible.

It would be exciting. :-)

Thank you,
Rick C. Hodgin

Draw backs of HPFS, long chkdsk time, hate to think how long a chdsk /f:3 on a 2TiB partition would take :) Lack of large file support, the 2GiB file size limit is a problem. It is possible to work around this with the DEADBEEF flag, which changes seeking from per byte to per sector with caller having to manage things.
I don't know much else about the structures.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #11 on: June 02, 2017, 02:37:12 am »
Draw backs of HPFS, long chkdsk time, hate to think how long a chdsk /f:3 on a 2TiB partition would take :) Lack of large file support, the 2GiB file size limit is a problem. It is possible to work around this with the DEADBEEF flag, which changes seeking from per byte to per sector with caller having to manage things.
I don't know much else about the structures.

I've been reading about this tonight.  It looks like large file support could be overcome, as the theoretical maximum is over 7+GB as is.  That 7+GB limitation could be defeated with a tweak it could be extended out as far as is needed.

It seems the only true limit right now to very large files in HPFS is in the legacy API, which only receives 32-bit value for file-related offsets.  That could also be defeated with our own open source kernel and new API which supports 64-bit file operations.

-----
If I had money, I would devote a team of developers to creating our own open source OS/2 clone from the ground up, kind of like what Mark Shuttleworth did with Ubuntu Linux, although he still built atop the community base, and ours would be a full ground-up replacement effort, but I would be supporting the full legacy OS/2 API so all of our code could be compiled on existing OS/2 code bases for testing, and would also compile on the new OS/2.

Ah to dream...

Thank you,
Rick C. Hodgin
« Last Edit: June 02, 2017, 02:48:46 am by Rick C. Hodgin »

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Thank You
  • -Given: 243
  • -Receive: 46
  • Posts: 1637
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: HPFS and JFS internal definition specifications
« Reply #12 on: June 02, 2017, 02:03:06 pm »
Hi

I restored the original fig5 and fig6 from the original source on the HPFS article. I don't what if the optimized images got list or maybe never was finished. I will check on that.

Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Rick C. Hodgin

  • Full Member
  • ***
  • Thank You
  • -Given: 12
  • -Receive: 2
  • Posts: 130
    • View Profile
    • Liberty Software Foundation
Re: HPFS and JFS internal definition specifications
« Reply #13 on: June 02, 2017, 02:07:14 pm »
I restored the original fig5 and fig6 from the original source on the HPFS article. I don't what if the optimized images got list or maybe never was finished. I will check on that.

Excellent.  Now that I see what they are, I'll go ahead and create some new optimized images and you can update them when completed.

Do you have an original source document for fig6?  It may be hard for me to decipher.

Thank you,
Rick C. Hodgin

Lars

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 20
  • Posts: 140
    • View Profile
Re: HPFS and JFS internal definition specifications
« Reply #14 on: June 02, 2017, 02:47:48 pm »
There already exists a "large file" API and there also exists additional entry points and extensions for existing entry points for file system drivers to support these new APIs.
These APIs will use 64-bit signed values to deal with file size and seek offsets.

The only problem is that the current HPFS.IFS was written at a point in time where these additional entry points where not yet defined.
If these additional entry Points were added to the existing HPFS.IFS then it would not be necessary to use the "DEADBEEF hack" and read in units of sectors instead of bytes.

As an example, there is an effort currently going on to add these additional "large file" entry points to the existing FAT32.IFS.
That will finally allow to deal with files up to 2^63 Bytes and likewise will allow byte granular accesses/seeks across disks up to a size of 2^63 bytes.


Lars


Draw backs of HPFS, long chkdsk time, hate to think how long a chdsk /f:3 on a 2TiB partition would take :) Lack of large file support, the 2GiB file size limit is a problem. It is possible to work around this with the DEADBEEF flag, which changes seeking from per byte to per sector with caller having to manage things.
I don't know much else about the structures.

I've been reading about this tonight.  It looks like large file support could be overcome, as the theoretical maximum is over 7+GB as is.  That 7+GB limitation could be defeated with a tweak it could be extended out as far as is needed.

It seems the only true limit right now to very large files in HPFS is in the legacy API, which only receives 32-bit value for file-related offsets.  That could also be defeated with our own open source kernel and new API which supports 64-bit file operations.

-----
If I had money, I would devote a team of developers to creating our own open source OS/2 clone from the ground up, kind of like what Mark Shuttleworth did with Ubuntu Linux, although he still built atop the community base, and ours would be a full ground-up replacement effort, but I would be supporting the full legacy OS/2 API so all of our code could be compiled on existing OS/2 code bases for testing, and would also compile on the new OS/2.

Ah to dream...

Thank you,
Rick C. Hodgin