Author Topic: JFS failing in a disastrous way - without the user knowing  (Read 22755 times)

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
JFS failing in a disastrous way - without the user knowing
« on: November 30, 2022, 01:01:18 pm »
Hallo,

I thought, I'd share this info with everybody.

Initial info:
0) as a standard case that is likely true for 99.9 % of all OS/2 systems in use today, my system/boot partition is JFS formatted
1) a long time ago, I had installed eCS as german NLS version. What that means for this error case is that some standard directories of the WPS are created with umlauts in their directory names, for example, "Desktop" becomes "Arbeitsoberfläche" in german.
2) as some of you might know, JFS.IFS saves its directory and file names in UNICODE and relies on the services of the UNICODE.SYS driver to properly convert between UNICODE and the default ASCII character set in use because effectively, the IFS interface only supports ASCII directory and filenames. The consequence of that is that you should NEVER hop between different character sets (by using the "chcp" command) when you create directories or files that have special characters in them (anything beyond the US437 character set).
3) for various reasons I have multiple boot configurations with different "config.sys" files, managed by the QSINIT selection menu. These files have different names: config.sys, config.smp, config.w4 etc.

Recently I decided to install this package:
https://hobbes.nmsu.edu/download/pub/os2/apps/wps/xworkplace/widgets/CPUTemp_wdgt_1-1-0.zip
That was the first error, because I have an AMD CPU and therefore this package was useless for me :-)

What followed next was the "cputemp" WPI package updating the default "config.sys" file by placing a driver into it. As it turned out later, it would not only add a driver load entry but also, apparently due to a bug in WarpIn (I later extracted the install script from the WPI file and found nothing wrong or suspicious in it), would remove the load line for UNICODE.SYS from the config.sys file. Because I had booted from a boot configuration that used another "config.sys" file (named, say "config.smp") this problem did not become apparent for as long as I was booting in the alternate boot configuration.

Eventually I booted into the standard boot config using file "config.sys". The base boot of loading all drivers and IFS (including JFS.IFS) went by without any problem but eventually, I was presented by the XWorkplace message that my desktop directory could not be found.
I was not too worried and suspected a simple OS2.INI or OS2SYS.INI corruption and decided to restore this set of files from a boot a few days earlier.

But on the next reboot, the error showed up again. From the commandline of the temporary desktop that had been created I decided to look directly at the filesystem naming of directories and files. And yes, "Arbeitsoberfläche" had been corrupted at the umlaut character. I then took the bad choice (in hindsight) and attempted to rename the directory back to what it was but it turned out that I was unable to do so. I was also unable to delete or rename that directory with any other tool like File Commander.
I decided to copy the desktop from a backup folder and was successful but intimidating enough, now I had two directories with the very same name "Arbeitsoberfläche" (on filesystem level !) but one of it could neither be renamed nor deleted. To remember: UNICODE.SYS driver was not loaded at this point.

It dawned on me that something had gone terribly wrong with the file system.Using DFSee, I triggered a chhdsk for the next reboot. The chkdsk would end without error but now I would directly run into a trap right after chkdsk was done. This was repeatable.

Fortunately, I had a completely different "development" boot partition that was completely intact (and both, UNICODE.SYS and JFS.IFS propery loading). And I had DFSee. But that was installed on the JFS partition that I could not boot from. Fortunately, as turned out later, that did not matter. The "broken" partition had directory naming problems but was intact as such and therefore I was able to readd the UNICODE.SYS load entry to "config.sys" on the "broken" partition.

I fired up the "development" boot partition and started DFSee. In one of its extended menus I found the possibility to rename directories and files on a very low file system level (so to say). Apparently, DFSee understands all low level structures of JFS and can directly overwrite sector content.

Subsequently it turned out that not only "Arbeitsoberfläche" had been "renamed" disastrously but also all OTHER DIRECTORIES below "Arbeitsoberfläche" that contained umlauts (yes there are quite a few directories below the "Desktop" directory ...).

For each of the affected directories, I had to hop back and forth between giving them an intermediate name like "Arbeitsoberflxxxx" and "Arbeitsoberfläche" to force the proper directory name (intermediate reboots required). After multiple reboots and some traps I was finally able to restore the partition to an acceptable state, but I had to recreate quite a few directories under the "Desktop" directory and move things to their proper place.


I think it would at least be helpful if JFS.IFS would, ideally already on bootup, report a warning if it fails in using UNICODE.SYS services instead of reporting nothing. That would have pointed me to the problem and would have saved me hours of work.
(I can understand that JFS.IFS will need to be able to continue nonetheless because otherwise you would not be able to fix the config.sys file and readd the UNICODE.SYS load entry ...).

This behaviour is surely an absolute NOGO for a new user (unless you only accept english speakers or people using the US/UK NLS version as new users).







Mike Kölling

  • Full Member
  • ***
  • Posts: 130
  • Karma: +6/-0
    • View Profile
Re: JFS failing in a disastrous way - without the user knowing
« Reply #1 on: November 30, 2022, 01:29:49 pm »
Hi Lars,

Thanks for your detailed explanation about the JFS issue without the unicode.sys driver.
As a German user of ANOS with a lot of Umlaute, because I renamed a lot of the originally English named folders to common German names, I think it might help me to avoid similar waste of time for finding out the root cause of the problem.

Thanks and Greetings from Potsdam / Germany,
Mike

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: JFS failing in a disastrous way - without the user knowing
« Reply #2 on: November 30, 2022, 02:11:56 pm »
Oh dude...I feel your pain!!!

1st of all, glad that you recovered, but yeah, as I read your post I likewise had the tell-tale feeling of "butterflies" in my stomach, old memories came back here.

A few years back I made the move from HPFS386 to JFS, but since this was NOT a fresh install I managed to convert stuff (standard XCOPY partition move approach, etc.) while UNICODE.SYS was NOT preceeding the JFS call in CONFIG.SYS. Bottom line being the NEW system was quite usable, appeared to work just fine, but soon enough strange things started to happen and eventually I had a major JFS problem on my hands.

In this case the root cause was my unfamiliarity with true JFS requirements and the fact that I missed tossing UNICODE.SYS into my CONFIG.SYS ahead of the JFS IFS call and ahead of the partition move process itself.

Your root cause is quite different from that, but at the time of my troubles I do recall thinking to myself: "well, if the darn thing is THAT important why isn't the IFS at least shouting - should really scream at me though - when it finds it missing?".

Andi B.

  • Hero Member
  • *****
  • Posts: 811
  • Karma: +11/-2
    • View Profile
Re: JFS failing in a disastrous way - without the user knowing
« Reply #3 on: November 30, 2022, 03:25:18 pm »
These Umlaut problem hits us OS/2 users since the beginning. The root of the problem is IMO IBMs decision to translate important folders to something with special characters (Arbeitsoberfläche). A lot of us where bitten by this ages before JFS arrived when we simply played with codepage settings in config.sys or CHCP. It's the same with HPFS(386). When you change the codepage setting in config.sys, chances a good that on the next reboot you're Desktop is lost/destroyed. Only difference is - we learned not to do this. Never ever play with codepage setting (at least the first one). We also have to learn not the delete unicode.sys when using JFS. Maybe Martin can delete these package from hobbes. Or when it's really a warpin issue then I think it is solved in current releases. I remember reading about that problem (deleting unicode from config.sys) a while ago.

Quote
because I renamed a lot of the originally English named folders to common German names
I hope you don't make this problem worse and don't use umlauts. Even if the meaning of the translation sounds not as exact as it could be, it's far better to use another word and circumvent any special character in (system relevant) file/folder names.
« Last Edit: November 30, 2022, 03:27:53 pm by Andi B. »

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: JFS failing in a disastrous way - without the user knowing
« Reply #4 on: November 30, 2022, 05:34:04 pm »
I ran into the same issue, excepting with a English install, there were just some MP3's with umlauts. In my case everything was fine, including the umlaut files being fine, until I booted to a different install, where those file names became garbage (actually I forget if it was umlauts or other special characters). Took a while to figure out it was a missing unicode.sys and once I did figure it out, I renamed the problem files without unicode.sys installed.
This is a known problem that was reported to Digi long ago and he fixed the problematic CONFIGSYS="DEVICE=?:\OS2\rdmsr.sys | UNIQUE" line, which upon not finding rdmr.sys, replaced the first line containing .sys, usually unicode.sys. There's an issue, but it is private. Remy filed it. Unluckily it was quite a while before it was uploaded to Hobbes.

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
Re: JFS failing in a disastrous way - without the user knowing
« Reply #5 on: November 30, 2022, 07:46:47 pm »
...
Subsequently it turned out that not only "Arbeitsoberfläche" had been "renamed" disastrously but also all OTHER DIRECTORIES below "Arbeitsoberfläche" that contained umlauts (yes there are quite a few directories below the "Desktop" directory ...).

For each of the affected directories, I had to hop back and forth between giving them an intermediate name like "Arbeitsoberflxxxx" and "Arbeitsoberfläche" to force the proper directory name (intermediate reboots required). After multiple reboots and some traps I was finally able to restore the partition to an acceptable state, but I had to recreate quite a few directories under the "Desktop" directory and move things to their proper place.


I think it would at least be helpful if JFS.IFS would, ideally already on bootup, report a warning if it fails in using UNICODE.SYS services instead of reporting nothing. That would have pointed me to the problem and would have saved me hours of work.
(I can understand that JFS.IFS will need to be able to continue nonetheless because otherwise you would not be able to fix the config.sys file and readd the UNICODE.SYS load entry ...).

This behaviour is surely an absolute NOGO for a new user (unless you only accept english speakers or people using the US/UK NLS version as new users).

Hi,

I had a similar issue after installing Lotus office program in French.
All directories having characters with accent where lost.
I have DFSee and with the help of the developper, I could find bad directories and correct them without use of a backup system.

In my case, they was a problem with unicode and my issue arrived after having put duplicated folder into the trash cam.

Of course, warning message would be nice in case of language with accent chars.   

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4713
  • Karma: +41/-1
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: JFS failing in a disastrous way - without the user knowing
« Reply #6 on: November 30, 2022, 08:25:02 pm »
I ran into the same issue, excepting with a English install, there were just some MP3's with umlauts. In my case everything was fine, including the umlaut files being fine, until I booted to a different install, where those file names became garbage (actually I forget if it was umlauts or other special characters). Took a while to figure out it was a missing unicode.sys and once I did figure it out, I renamed the problem files without unicode.sys installed.
This is a known problem that was reported to Digi long ago and he fixed the problematic CONFIGSYS="DEVICE=?:\OS2\rdmsr.sys | UNIQUE" line, which upon not finding rdmr.sys, replaced the first line containing .sys, usually unicode.sys. There's an issue, but it is private. Remy filed it. Unluckily it was quite a while before it was uploaded to Hobbes.

Hi

Dave. I didn't get right if the bug on WarpIn was fixed or not?
Lars, was this an specific version of WarpIn, or was it with the latest one (1.0.26)?

Regards
« Last Edit: November 30, 2022, 08:50:29 pm by Martin Iturbide »
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4713
  • Karma: +41/-1
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: JFS failing in a disastrous way - without the user knowing
« Reply #7 on: November 30, 2022, 08:31:31 pm »
Hi

I quickly installed "CPUTemp_wdgt_1-1-0.zip" with Warpin 1.0.24 (ArcaOS 5.0.6) and I confirm it removed UNICODE.SYS from my config.sys.

Reverted everything, upgraded to WarpIn 1.0.26 and I got the same thing. UNICODE.SYS gets removed.

I also don't see anything strange on the CPUTEMP WarpIn script, just the "CONFIGSYS="DEVICE=?:\OS2\rdmsr.sys | UNIQUE" which should not remove UNICODE.SYS.

Should it be reported as WarpIn bug?

Regards
« Last Edit: November 30, 2022, 08:50:11 pm by Martin Iturbide »
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4713
  • Karma: +41/-1
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: JFS failing in a disastrous way - without the user knowing
« Reply #8 on: November 30, 2022, 09:05:49 pm »
Hi

I just find out (thanks to this thread) that Digi updated the CPUTemp plugin to version 1.1.2 on the "System Load" package.
I think he uses a different script to try to avoid this bug:
 EXECUTE="CONFIGSYS | $(102)\drvins.cmd | -d ?:\ -f $(102)\rdmsr.sys -i 1"
 DEEXECUTE="$(102)\drvins.cmd | -d ?:\ -f $(102)\rdmsr.sys -i 0"

Anyways, I think I will be removing "CPUTemp_wdgt_1-1-0.zip" from hobbes, because System Load has a newer version.

Regards
 
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: JFS failing in a disastrous way - without the user knowing
« Reply #9 on: November 30, 2022, 10:01:34 pm »
Hi

I quickly installed "CPUTemp_wdgt_1-1-0.zip" with Warpin 1.0.24 (ArcaOS 5.0.6) and I confirm it removed UNICODE.SYS from my config.sys.

Reverted everything, upgraded to WarpIn 1.0.26 and I got the same thing. UNICODE.SYS gets removed.

I also don't see anything strange on the CPUTEMP WarpIn script, just the "CONFIGSYS="DEVICE=?:\OS2\rdmsr.sys | UNIQUE" which should not remove UNICODE.SYS.

Should it be reported as WarpIn bug?

Regards

Warpin is working as it should based on discussion on the testers list. What happens is if rdmsr.sys is not installed, it matches up with the first *.sys in config.sys and replaces that with rdmsr.sys. Usually it is unicode.sys at the top of config.sys.

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4713
  • Karma: +41/-1
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: JFS failing in a disastrous way - without the user knowing
« Reply #10 on: December 01, 2022, 12:27:50 am »
Hi Dave

Checking the WarpIn wpi_prg.inf file.
Code: [Select]
UNIQUE[(statement2)]
This makes sure that the statement part of the CONFIGSYS attribute occurs only once, i.e. will be unique in CONFIG.SYS, by searching for a similar line and replacing that line with the new line. If no such similar line is found, the new line is appended to the bottom of CONFIG.SYS (that position can be changed with ADDTOP, ADDBEFORE, ADDAFTER ).

All searching is case-insensitive.

This assumes that statement contains a "=" character, as all CONFIG.SYS statements do. The exact behavior of UNIQUE depends on the syntax:

If UNIQUE is specified all alone (without the brackets and statement2), CONFIG.SYS is searched for a line containing the part before the "=" char of statement.

This is only useful for single CONFIG.SYS statements such as SET xxx= whose left part (before "=") will occur only once in CONFIG.SYS.

Example: CONFIGSYS="PROTECTONLY=YES | UNIQUE" would replace any "PROTECTONLY=" statement with the new line. If no such line exists, a new line is added to the bottom (that position can be changed with ADDTOP, ADDBEFORE, ADDAFTER).

If UNIQUE is specified with the (statement2) part, CONFIG.SYS is searched for a line which contains the part before the "=" character of statement and statement2 after the "=" character as a substring.

Example:

CONFIGSYS="BASEDEV=DANIS506.ADD | UNIQUE(IBM1S506.ADD)"

would search for a line containing both BASEDEV= and IBM1S506.ADD and replace that line with BASEDEV=DANIS506.ADD.

This syntax is required for adding device drivers, because there will surely be more than one BASEDEV statement in CONFIG.SYS.

This also works with the DEVICE= statements which can have a full path specification, as follows:

CONFIGSYS="DEVICE=$(1)\bin\crashsys.drv | UNIQUE(stable.drv)"

This would replace a line containing both DEVICE= and stable.drv with DEVICE=[path_of_package_1]\bin\crashsys.drv, using macro resolution.

Do you mean that it should be used on the WarpIn script like...
  "CONFIGSYS="DEVICE=?:\OS2\rdmsr.sys | UNIQUE(rdmsr.sys)"
to avoid getting the Unicode.sys replaced?

Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: JFS failing in a disastrous way - without the user knowing
« Reply #11 on: December 01, 2022, 01:22:21 am »
Do you mean that it should be used on the WarpIn script like...
  "CONFIGSYS="DEVICE=?:\OS2\rdmsr.sys | UNIQUE(rdmsr.sys)"
to avoid getting the Unicode.sys replaced?


I'm not an expert with Warpin, it seems that would still not add rdmsr.sys if it wasn't in config.sys to begin with but would avoid replacing unicode.sys. Guess that is why Digi ended up with a bit of Rexx to do it.

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4713
  • Karma: +41/-1
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: JFS failing in a disastrous way - without the user knowing
« Reply #12 on: December 01, 2022, 02:10:33 pm »
Thanks for the feedback.

The second thing.
Will it makes sense to request Arca Noae to put a "Boot Warning Text" for JFS.IFS if it does not find UNICODE.SYS loaded ? Does it makes sense? Can it be possible in boot time to detect this for JFS.IFS?

Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
Re: JFS failing in a disastrous way - without the user knowing
« Reply #13 on: December 01, 2022, 02:21:24 pm »
Thanks for the feedback.

The second thing.
Will it makes sense to request Arca Noae to put a "Boot Warning Text" for JFS.IFS if it does not find UNICODE.SYS loaded ? Does it makes sense? Can it be possible in boot time to detect this for JFS.IFS?

Regards

Does it have sense to have unicode if language is EN only ?

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: JFS failing in a disastrous way - without the user knowing
« Reply #14 on: December 01, 2022, 04:22:45 pm »
Thanks for the feedback.

The second thing.
Will it makes sense to request Arca Noae to put a "Boot Warning Text" for JFS.IFS if it does not find UNICODE.SYS loaded ? Does it makes sense? Can it be possible in boot time to detect this for JFS.IFS?

Regards

Yes, that would definitely make sense.

I would need to check the OpenJFS source code, but I would think that JFS.IFS uses the KEE API calls "KernCreateUConvObject", "KernStrToUCS" and "KernStrFromUCS" to create a UConv object, convert from an ASCII string to a Unicode string (taking into account active system codepage), convert from a Unicode string to an ASCII string, respectively.

I would suspect that "KernCreateUConvObject" is called very early, in the IFS loading/initialization stage (but not already on FS_INIT as that executes in Ring 3 whereas calling "KernCreateUConvObject" requires executing in Ring 0). The call will definitely return with an error if UNICODE.SYS is not loaded (these KEE API calls are the ones that require UNICODE.SYS to be loaded !!!) and that would be the indirect way to find out.

If it is not possible to already display a message on boot, then there are options:
1) have a daemon executing that could show a warning message. But that requires to implement a daemon interface in JFS.IFS and also to start the daemon application itself which of course can also go wrong.
2) Another, potentially better option is to write to the system log (the syslog.exe thing that nobody ever uses ...) because that would be a persistent logging (logging happens to a file) but that in turn also requires an additional device driver and daemon (log.sys and logdaem.exe) when the logging is to be done from a device driver or IFS. I also do not know if that conflicts with the role of the IFS giving access to files (if log.sys buffers to memory and logdaem.exe writing much later in the process, then that could work). In that case, AN should ensure that LOG.SYS and LOGAEM.EXE are always added to config.sys on each install.
3) last but not least to use the low level file system device driver calls to write a file in the root directory (say with fixed string: "Attention: UNICODE.SYS NOT LOADED !"). I guess that is the best option as it requires no further components to be up and running. But I seem to remember that also requires ring 0 execution which again would mean, you'd need to delay the logging until the first IFS call executing in Ring0 and also after the filesystem has been mounted (obviously).

But the problem should be solvable.