Author Topic: WCD broken when running against JFS???  (Read 18995 times)

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
WCD broken when running against JFS???
« on: July 26, 2018, 06:59:06 pm »
Ugghhh????!!!  :o

Yeah...strange right?

So I am literally coming out of a move from my multi-HPFS386-partition setup to a much simpler and (hopefully) more efficient JFS partition setup.

Since all HPFS partitions are limitted to 64Gig it was just getting to be too painfull watching the system boot, let alone the occasional CHKDSK execute here and there. In combination I also deployed a SSD drive (Samsung Evo850 - 250Gig) to replace my aging WD Raptor (BTW: if you have not thought about it...I am telling you, there is no computing like "SSD Computing" LOL...that sweet!!!).

Anyways, so I managed to get the partitions migrated (xcopy, sysinstx, etc), all came up fine. I replaced multiple HPFS386 partitions with a single JFS boot partition. So I went from multiple 64Gig to a single 233Gig drive.

Now here is where it gets weird. I used to run WCD (RPM 6.0.2, 2018-05-10 drop) very successfully in the past. Since I got rid of a pile of HPFS386 partitions I figured I would re-do the WCD scan and get it down to just my single JFS partition...lo and behold, the 'WCD -S g:\' CLI causes one of my CPU cores to immediately spike to 100% use, RAM to be consumed (pretty fast too) and eventually producing a trap:

Code: [Select]
[G:\usr\bin]wcdos2 -S g:\
Wcd: Please wait. Scanning disk. Building treedata-file g:/home/wcd/treedata.wcd
 from g:/

Killed by SIGSEGV
pid=0x0068 ppid=0x0065 tid=0x0001 slot=0x00e2 pri=0x0200 mc=0x0001 ps=0x0010
G:\USR\BIN\WCDOS2.EXE
LIBC066 0:0004cecb
cs:eip=005b:1d6dcecb      ss:esp=0053:00139d94      ebp=0013a058
 ds=0053      es=0053      fs=150b      gs=0000     efl=00010212
eax=00000000 ebx=ab030000 ecx=ab030000 edx=2003013c edi=00000009 esi=00139ecd
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

Before I log a ticket (because this is really weird, I have never seen anything like this before), has anyone experienced this?

Pete

  • Hero Member
  • *****
  • Posts: 1281
  • Karma: +9/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #1 on: July 29, 2018, 01:47:54 am »
Hi Dariusz

I must admit to never having used wcd myself but the trap looks libc066 related. Is your "unix system" up to date with no older copies hanging around?

I note that you have "... replaced multiple HPFS386 partitions with a single JFS boot partition." which I do not think is a good idea. My own experience of JFS boot volumes is that they can be somewhat unreliable - and somewhere on the Arca website HPFS rather than JFS is recommended for boot volumes. My own preference is to use small (2Gb is more than adequate) HPFS boot volumes and have apps/data on larger JFS volumes. Just my thoughts...


Regards

Pete





Doug Bissett

  • Hero Member
  • *****
  • Posts: 1593
  • Karma: +4/-2
    • View Profile
Re: WCD broken when running against JFS???
« Reply #2 on: July 29, 2018, 03:47:00 am »
I agree that boot volumes should be small (about 2 GB, is more than adequate), and programs/data should be on a different drive. I do NOT agree that the boot volumes should be HPFS. I find that JFS works remarkably well for boot drives. Far better than HPFS ever did (especially when a boot time CHKDSK is required, which is almost "never"). The only thing that I use HPFS for is the RAMDISK (only because it is more difficult to convert it to JFS, and I can make the HPFS cache very small to minimize the adverse effect of using a cache with a RAMDISK).

As for WCD, I haven't used it, so I really can't comment. If Peter's suggestion that the problem is LIBC related, the latest version seems to have fixed a number of problems.

Somebody, somewhere, also suggested adding the line:
Code: [Select]
SET LIBCX_HIGHMEM=2to CONFIG.SYS. As Peter said, make sure that you don't have stray copies of the RPM/YUM stuff laying around. Give them half a chance, and they will cause problems.

Andy Willis

  • Sr. Member
  • ****
  • Posts: 292
  • Karma: +7/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #3 on: July 29, 2018, 07:11:28 pm »
I've never used wcd either, I had built a REXX changdir years ago and then once I started using 4os2 I largely use the built in cdd.  Any how, I downloaded the WCD 5.3.0 builds from hobbes.
The only reason I have HPFS at this point is like Doug, my RAMDISK is HPFS, in my case because I don't know any way to format it JFS when I have my swapper.dat on it.  I couldn't set the /cache for HPFS less than 8 (otherwise it defaults to 64) but in startup.cmd I turn off the hpfs lazy and readahead cache. 
I do not have the sigsegv in wcdos2.  However, I do see a major difference between the gcc and watcom builds in the treedata.wcd.
The gcc built wcdos2 adds the first 11 directories, then it skips some thousands of directories and then picks up e:\usr\bin and the one directory I have under e:\usr\bin.  So all in all it picks up 13 directories.  It does this very very quickly.
The watcom build wcdos2 runs much much longer and seems to pickup every directory in treedata.wcd (175598 directories listed).
All in all the gcc built version behaves quite oddly here.

Silvan Scherrer

  • Full Member
  • ***
  • Posts: 200
  • Karma: +1/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #4 on: July 30, 2018, 10:58:21 am »
Which wcd build it the gcc one? The one from RPM? As imho the most current one is 6.0.2.
kind regards
Silvan
CTO bww bitwise works GmbH

Please help us with donations, so we can further work on OS/2 based projects. Our Shop is at https://www.bitwiseworks.com/shop/index.php

waterlan

  • Guest
Re: WCD broken when running against JFS???
« Reply #5 on: July 30, 2018, 11:43:21 pm »
... Any how, I downloaded the WCD 5.3.0 builds from hobbes.
...
I do not have the sigsegv in wcdos2.  However, I do see a major difference between the gcc and watcom builds in the treedata.wcd.
The gcc built wcdos2 adds the first 11 directories, then it skips some thousands of directories and then picks up e:\usr\bin and the one directory I have under e:\usr\bin.  So all in all it picks up 13 directories.  It does this very very quickly.
The watcom build wcdos2 runs much much longer and seems to pickup every directory in treedata.wcd (175598 directories listed).
All in all the gcc built version behaves quite oddly here.

This problem was fixed in wcd 6.0.0.
The problem was caused by wcd not handling emx symbolic links correctly.
The Watcom build just ignores them.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #6 on: August 05, 2018, 05:24:02 pm »
Hi Pete,

...I must admit to never having used wcd myself but the trap looks libc066 related. Is your "unix system" up to date with no older copies hanging around?...

Well, as best as I can tell I am throughly YUM/RPM based, meaning, all the individual DLL based stuff has been migrated to the RPM versions where such releases have become available. The "unix system" appears to be fully functional in my install.

...I note that you have "... replaced multiple HPFS386 partitions with a single JFS boot partition." which I do not think is a good idea. My own experience of JFS boot volumes is that they can be somewhat unreliable - and somewhere on the Arca website HPFS rather than JFS is recommended for boot volumes. My own preference is to use small (2Gb is more than adequate) HPFS boot volumes and have apps/data on larger JFS volumes. Just my thoughts...

This situation is a legacy of my multiple years of OS/2 use. I have always had a single main OS/2 partition where all apps as well as the underlying OS lived. Eventually as the 64Gig HPFS limit was reached I moved my DATA stuff (media, docs, etc) to other partitions. With bootable JFS now available I had a choice of either re-installing all this stuff as per the recommended separate OS & separate APPS structure, or keeping things AS-IS and proceeding. Needless to say, I was looking for the least amount of change (no time) and so kept the AS-IS approach.

You are absolutely right about JFS not being quite ready for prime-time...I'm seeing this first hand, some strange results here, such as for example during boot-up I'm now back to the dreaded WPS 'blue-screen' freeze. Setting JFS cache to something like 128MB occasionally causes my display to go to power saving mode during boot (SNAP video drivers here) and I have to CTL-ALT-DEL. The whole JFS tuning topic is a bit of 'black magic', I've been looking for OS/2 reference info and so far beyond a couple of Warpstock presentations I've only found references to some attempts to port the OpenJFS to our platform.

Yeah, I loved HPFS386 and if it wasn't for that pesky 32-bit limit I'd happily stick with it. But having multiple partitions, lengthy boot-up times, etc, etc just proved to be too much. My thinking is: if the tech you are using ends up being a hinderance, maybe it's time has come and you must look for alternatives? JFS is that alternative I am currently pursuing...

ivan

  • Hero Member
  • *****
  • Posts: 1558
  • Karma: +17/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #7 on: August 05, 2018, 07:07:51 pm »
Dariusz, I feel for you but I too wonder about the wisdom of putting everything on one JFS partition.  I learned years ago the consequences of doing that but fortunately I only had to reformat one partition and the backup was up to date.

When I first started using OS/2 (v2.1) one of the techies at the local IBM office advised me to use at least two partitions - one for the operating system and the other for programs and data - The reason he gave was that if a fixpak messed up the OS I could reinstall without touching the programs and data.

Over the years those two partitions have turned into 7 - OS (on a small HPFS), Tools, Programs, Work, Home (a concession to the ported programs), General (for things like mozprofiles etc.) and Store (where items are stored as it is generally quicker to get than going to one of the NAS boxes). Except for the OS partition they are all standard JFS partitions.

Lately I have been experimenting with AMD ITX boards with APU second gen A series and Ryzen processors. I have also decided to use FireCuda 2 Tb disks as standard.  With those disks the partitioning is setup using DFSee (disks plugged into a SATA/USB adapter).  I can then format them and copy what I need over. I do allow ArcaOS to format the OS partition as HPFS and copy files to the correct partitions, this is a continuation of what I did before with OS/2.

Why am I rattling on?  It might give you some ideas, it might help others as others have helped me in the past.  I will also add that doing it this way has cut down the recovery time after a total disk failure to half an hour (we made images of the basic data on the partitions).
 

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #8 on: August 06, 2018, 12:08:19 am »
Hey Doug,

...I find that JFS works remarkably well for boot drives. Far better than HPFS ever did (especially when a boot time CHKDSK is required, which is almost "never"). The only thing that I use HPFS for is the RAMDISK (only because it is more difficult to convert it to JFS, and I can make the HPFS cache very small to minimize the adverse effect of using a cache with a RAMDISK)...

The is precisely my goal in the current conversion. Other then the issued I highlighted in my previous post, so far the JFS stuff appears to work OK and is without a doubt a mile ahead of what the HPFS396 could deliver given today's hardware.

So specifically, here is the storage definition of what I have today:

Code: [Select]
Filesystem                             Size  Used Avail Use% Mounted on
MAINT                                   54M   22M   33M  40% F:/
OS2                                    233G   82G  151G  36% G:/
MAINT_BKUP                              54M   22M   33M  40% H:/
OS2_BKUP                               233G   61G  173G  26% I:/
SPARE                                   47G  6.0M   47G   1% J:/
DATA_STORE                             466G   40G  427G   9% K:/
SMBFS64 \\HOME:winmain\C@guest         932G  163G  770G  18% T:/
SMBFS64 \\HOME:orclsrvr\users@dariusz  559G  351G  209G  63% U:/
SMBFS64 \\home:nas@admin                16Z  926G -1.8T    - V:/
                                       4.8G   12M  4.8G   1% Y:/

The XXX_BKUP are the hot backup partitions, I used to use a nightly dsync copy but with JFS dsync fails on my main boot partition (it is OK for the other ones though...???), so I'm learning about rsync to use that instead.

The fair 'working set' partion setup is: MAINT, OS2, SPARE, DATA_STORE, everything else is a backup.

BTW: something else I just learned today, the plain HPFS (2M cache) setup for the ramdisk (Y: in the above) is way faster than the HPFS386 (4M cache) one...I mean the SysBench throughput is 3-4x faster. Not that it matters much I suppose, but still, 403615.991 File I/O-marks on HPFS vs 163755.776 File I/O-marks on HPFS386. I simply did NOT expct to see that.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #9 on: August 06, 2018, 12:17:41 am »
Hi Ivan,

Dariusz, I feel for you but I too wonder about the wisdom of putting everything on one JFS partition....

Today I absolutely agree with you. The thing that's going to make me undertake a full re-install is when the AOS versions have gone through enough releases and testing that it is deemed positively capable of handling the newer hardware. Until that happens I am shying away from having to take on that migration task.

...Lately I have been experimenting with AMD ITX boards with APU second gen A series and Ryzen processors...

This is actually the bigger goal on my part. The older AMD Phenom X6 II hardware I'm currently on has been a great setup, but it's showing it's age. Ryzen II is my next stopping point. I've been closely watching the feedback on these (such as yours) and will probably build a completely new hardware platform when I'm ready to try OS/2 on that new stuff. What I have today (the migration from HDD to SSD) is really a mid-point, although there is no denying that the speed increase from SSD is considerable and well worth the investment in $$$ and time. Heck, the SSD prices have come down significantly anyways, so a suggestion I would make to anyone still running HDD on our platform is to look at what SSDs have to offer.

ivan

  • Hero Member
  • *****
  • Posts: 1558
  • Karma: +17/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #10 on: August 06, 2018, 01:34:18 am »
This is actually the bigger goal on my part. The older AMD Phenom X6 II hardware I'm currently on has been a great setup, but it's showing it's age. Ryzen II is my next stopping point. I've been closely watching the feedback on these (such as yours) and will probably build a completely new hardware platform when I'm ready to try OS/2 on that new stuff. What I have today (the migration from HDD to SSD) is really a mid-point, although there is no denying that the speed increase from SSD is considerable and well worth the investment in $$$ and time. Heck, the SSD prices have come down significantly anyways, so a suggestion I would make to anyone still running HDD on our platform is to look at what SSDs have to offer.

The two stumbling blocks with getting OS/2 - ArcaOS working on modern (no more than 9 months old) AMD boards is total lack of USB 3.0/3.1 (xhci) drivers (that applies to Intel boards as well) and a modified OS2AHCI.ADD to allow working with the ASMedia AHCI chipset (the secondary AHCI interface on all Ryzen chipsets is based on the ASMedia chip).

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1317
  • Karma: +26/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #11 on: August 19, 2018, 03:33:43 pm »
One more data-point, I'm suspecting this is not just a WCD issue. I'm seeing the same problem with a QT application, that being QDiskUsage as well as a Java app, that being JDiskReport.

Both of them scan your HD, creating an INDEX file, which is then used to report the disk space usage by specific filetypes. The apps use slightly different interfaces and of course their functionality is somewhat different.

When executing the QT utility memory is rapidly consumed, the very same behaviour as with WCD. The Java program halts after just a few seconds of use, attempting to CANCEL gracefully goes nowhere though.

I will review the DLLs that both programs rely on, maybe there is something obvious there. If nothing pops out then I suspect maybe JFS itself is the issue?

EDIT
====
Not all partitions are seemingly affected. I can successfully do a scan of my MAINT partition...but that one is only 54M in size.
« Last Edit: August 19, 2018, 03:38:21 pm by Dariusz Piatkowski »

Andy Willis

  • Sr. Member
  • ****
  • Posts: 292
  • Karma: +7/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #12 on: August 20, 2018, 03:36:07 am »
I have 2 - 512G SSD drives, each with a 400G JFS partitions on them.  Now that I have installed WCD via yum I have no issues running it.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: WCD broken when running against JFS???
« Reply #13 on: August 20, 2018, 05:09:49 am »
Dariusz, it might be a good idea to run chkdsk from your maintenance partition on the partitions that are causing you problems. Perhaps a directory is recursively linking to itself. Could also check it with DFSee.

guzzi

  • Sr. Member
  • ****
  • Posts: 331
  • Karma: +0/-0
    • View Profile
Re: WCD broken when running against JFS???
« Reply #14 on: August 21, 2018, 01:24:39 am »
This may be completely unrelated, but I have had problems with filenames created under a different codepage than the ones currently in use. Changing into a directory containing filenames with invalid characters with FC/2 caused it to endlessly scan the directory and become unkillable. You can check this by issuing ctrl-q in the root of a drive in FC/2 on any subdirectory. If it hits a filename with invalid characters the file count will keep going up endlessly.