Author Topic: Troubleshooting process trap/dump...how?  (Read 1403 times)

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1436
  • Karma: +28/-0
    • View Profile
Troubleshooting process trap/dump...how?
« on: May 05, 2025, 05:51:12 am »
So I've got a weird problem that just seemingly has shown up about a couple of weeks ago: I've seen three separate RSYNC jobs die part-way through a LOCAL backup, and as best as I can tell my OS/2 box reboots at the same time.

Now the strange part is that I can't find anything listed in places like POPUPLOG.OS2 or as TRP files...so it's rather tough to figure out whether this is a hardware or a software issue. At the moment I am leaning towards this being a hardware thing.

Having said that, I know that this is not a CPU cooling problem as I am running a pretty capable setup here and that's something I've been through multiple times so I know there is spare room there...but it could be something else, such as memory, or even CPU/core problems.

Sooo...I went back to my CONFIG.SYS and re-enabled the system LOG stuff I had previously enabled:

Code: [Select]
...
DEVICE=G:\OS2\LOG.SYS /OFF
RUN=G:\OS2\SYSTEM\LOGDAEM.EXE /E:g:\tmp\trp
DUMPPROCESS=G:\tmp\trp /F=10240
RUN=G:\OS2\EPWDDR3.EXE
DEVICE=G:\OS2\EPWDD.SYS
...

Not sure though if there is anything else I should be looking at to try to maximize the data capture should RSYNC cause either that hardware or perhaps software fault again.

What have you used in your previous attempts to narrow this kind of a problem down?

Thanks!

Dave Yeo

  • Hero Member
  • *****
  • Posts: 5400
  • Karma: +128/-1
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #1 on: May 05, 2025, 06:56:38 am »
Sometimes reseating the ram and any other cards fixes these types of issues as wears off any oxidation.

Pete

  • Hero Member
  • *****
  • Posts: 1406
  • Karma: +13/-0
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #2 on: May 05, 2025, 07:23:03 am »
Hi Dariusz

Try reseating ram and any cards, disconnect and reconnect drive and power cables .

If that does not help: How old is PSU and when did you last open it and give it a clean?
Had a problem a few years ago where lengthy disk access caused system hangs. Took a few hours but traced that to a dying PSU - dust buildup had caused overheating. Replaced PSU and all worked fine.


Regards

Pete

Lars

  • Hero Member
  • *****
  • Posts: 1431
  • Karma: +72/-0
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #3 on: May 05, 2025, 08:06:04 am »
Hi Dariusz,

you are not alone with the "my system suddenly reboots without any trace" problem. I have experienced the same more than once.

1) you are using a very outdated version of FFST (RUN=G:\OS2\EPWDDR3.EXE, DEVICE=G:\OS2\EPWDD.SYS) that predates the stuff that came with  Warp 4 (and after, of course). Do not do that. I suggest to deinstall it. >= Warp 4 comes with everything you need.

2) Lower your VIRTUALADDRESSLIMIT value. I suggest to start with 512 (which is the lowest value and equivalent to having no high memory at all) and then, raise up to a value that is still lower than your current value. If 512 is too low, you will need to start with a value that works for your system. VIRTUALADDRESSLIMIT can only be raised in chunks of 512 MB (0x20000000). XWorkplace kernel object will only select values that are possible. The default value for VIRTUALADDRESSLIMIT is 1024 (MB).

My suspicion is that 2) is the most likely culprit. The problem with a too high value is that OS/2 seemingly can run into problems on memory allocation (be it the system arena located above the VIRTUALADDRESSLIMIT value, or the user area below this value). And OS/2 does a very bad job of detecting this problem with the result that it does not even stop with a critical error message and a system trap dump. Obviously, the failure to allocate memory in the system arena is a fatal problem that renders the system completely unusable.
An alternative behaviour to these sudden reboots seems to be a complete system freeze (with the same effect as if all HW interrupts are being disabled).

Lowering this value has still not completely eliminated the problem for me but it has lowered the frequency with which these sudden reboots occur.
« Last Edit: May 05, 2025, 08:29:47 am by Lars »

Neil Waldhauer

  • Hero Member
  • *****
  • Posts: 1124
  • Karma: +30/-0
    • View Profile
    • Blonde Guy
Re: Troubleshooting process trap/dump...how?
« Reply #4 on: May 05, 2025, 03:24:05 pm »
Rsync would crash here, too. I was doing a pretty large network rsync, but reliability improved greatly when I removed a lot of EA from my files. I used eaclean to remove the unnecessary UID, GID, MODE, INO, RDEV, GEN, FLAGS EAs. A side benefit was that my rsync ran several times quicker.
Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1436
  • Karma: +28/-0
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #5 on: May 05, 2025, 04:39:14 pm »
Dave and Pete,

Gents, I'm thinking what you pointed out should not be the root cause simply due to the fact that I have about 3 monts on this box since my most recent CPU change, and even less so given the tuning I executed thereafter (memory, etc.). So I would say that the "dust has barely settled", but...point noted, I regularly (every three months or so) clean the various air filters on the case, so I will certainly review all of the above!

Neil,
Understand what you are saying, but what's weird is that I've been using RSYNC for a few years now to routinely do this nightly backup. Sure, RSYNC has had other issues (some of what you pointed to), but never this kind of a re-boot. I suspect RSYNC however is putting on some stress on a part of the system (could be hardware, could be software) that's most likely failing.

Both of the above suggestions have sent me on a bit of a manual maintenance run: CHKDSK all my drives (JFS), etc...about to start all of that actually!

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1436
  • Karma: +28/-0
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #6 on: May 05, 2025, 05:07:17 pm »
Hi Lars,

1) you are using a very outdated version of FFST (RUN=G:\OS2\EPWDDR3.EXE, DEVICE=G:\OS2\EPWDD.SYS) that predates the stuff that came with  Warp 4 (and after, of course). Do not do that. I suggest to deinstall it. >= Warp 4 comes with everything you need.

OK, so I need a little help here. The OS/2 Toolkit doc "Problem Determination Programmer's Guide" has a decent amount of details on how to USE the SYSTEM and PROCESS dumps, logs, etc...but nowhere does it actually spell out the required pieces that are needed to set this up.

In fact, the complete "PROC DUMP" section of my CONFIG.SYS is as follows:

Code: [Select]
REM ******************************************
REM ***       OS/2 DUMP/TRACE - START      ***
REM ******************************************
DEVICE=G:\OS2\LOG.SYS /OFF
RUN=G:\OS2\SYSTEM\LOGDAEM.EXE /E:g:\tmp\trp
DUMPPROCESS=G:\tmp\trp /F=10240
RUN=G:\OS2\EPWDDR3.EXE
DEVICE=G:\OS2\EPWDD.SYS

REM RUN=G:\OS2\EPWROUT.EXE 1
REM RUN=G:\OS2\EPW.EXE

REM *** USB LARS DEBUG => 2 TRACEBUF lines below
REM TRACEBUF=128 /M=WRAP,QUEUED,NODTI /D=ALL
REM TRACE=ON 225,226
REM ******************************************
REM ***       OS/2 DUMP/TRACE - END        ***
REM ******************************************

...where I actually have all of the above REM'ed out normally and unless I hit an issue I want to track I never have this turned on.

The only mention of FFST I have is in the Toolkit doc, stating the following:

Quote
...The OS/2 Logging Facility device driver LOG$ is not loaded.  To activate logging enter DEVICE=\OS2\LOG.SYS and RUN=\OS2\SYSTEM\LOGDAEM.EXE in config.sys and reboot...

But at least that would imply having those two alone in CONFIG.SYS should be sufficient. I did go back to re-reading the notes and will remove the '/OFF' parameter of the LOG.SYS driver.

Strangely enough I have found no place on my machine where any FFST install/uninstal options exist. The standard 'Selective Uninstall' shows nothing there other than standard Warp4 stuff.

...
2) Lower your VIRTUALADDRESSLIMIT value. I suggest to start with 512 (which is the lowest value and equivalent to having no high memory at all) and then, raise up to a value that is still lower than your current value. If 512 is too low, you will need to start with a value that works for your system. VIRTUALADDRESSLIMIT can only be raised in chunks of 512 MB (0x20000000). XWorkplace kernel object will only select values that are possible. The default value for VIRTUALADDRESSLIMIT is 1024 (MB)...

I will go to this option as the last resort simply due to the fact that my RSYNC jobs have been executing on nightly basis for several years now, no misses.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 5400
  • Karma: +128/-1
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #7 on: May 05, 2025, 05:37:47 pm »

Lars

  • Hero Member
  • *****
  • Posts: 1431
  • Karma: +72/-0
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #8 on: May 05, 2025, 06:12:12 pm »
1) you will need LOGDAEMON.EXE so that (Ring 3) applications can use the "LogAddEntries" API to write to the System Log. Not many apps need it but it does not hurt to have it running. "LogAddEntries" API is only documented in the Debugging Handbook.

2) you will need LOG.SYS so that Ring 0 device drivers/IFS etc. can use the DevHlp_LogEntry device helper to write to the System Log. It is also required by LOGDAEMON.EXE to operate. Not many device drivers use it (if any) but it does not hurt to have it running

3) You can safely remove all that EP... stuff as it is the outdated FFST stuff which has been superseded long ago, see here:
https://www.edm2.com/index.php/First_Failure_Support_Technology/2
From the list, only keep the following as actually, that is the new FFST stuff (I will need to recheck just to be sure):
FFST.EXE
FFSTCONF.EXE
FFSTPCT.EXE
ERRLOG.DLL
FFCONFIG.DLL
FFDUMP.DLL
FFGUI.DLL
FFIPC.DLL
FFPCT.DLL
FFPROBE.DLL
FFST.DLL
FFST.HLP

4) FFST.EXE is the system log daemon, it is started by SMSTART.EXE (see what THESEUS.EXE reports). It seemingly replaces EPWDDE3.EXE, EPWROUT.EXE, EPW.EXE.

5) Finally, FFST.DLL contains routine "FFSTProbe" which effectively supersedes "LogAddEntries" (see above) to write to the System Log but both still coexist where "LogAddEntries" is easier to use but requires a handwritten DLL to format the logged data where "FFSTProbe" uses a text template formatting approach that however is overly complicated.


« Last Edit: May 06, 2025, 11:37:27 am by Lars »

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1436
  • Karma: +28/-0
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #9 on: May 06, 2025, 08:11:44 pm »
OK, so a bit of an update, and this may ultimately be a RSYNC issue, or perhaps a samba client problem...?

Last night the machine died...TWICE...lol, within 10 mins of each reboot. As it happened my cron schedule has multiple RSYNC jobs scheduled (they do NOT run in parallel though) and the 1st reboot happened in the middle of the 1st RSYNC, and the 2nd during the next scheduled run (different SOURCE & TARGET locations though, but both copying from a LOCAL directory to the NAS box - which is accessed through a NetDrive samba share).

So seeing as I had just about touched on all the other things I could possibly think of, I decided to run the same RSYNC job manually (as in not scheduled).

The RSYNC CLI is this:

RSYNC --archive --delete --progress --stats --whole-file --no-xattrs --human-readable --quiet  --exclude-from=G:\mptn\etc\rsync_exclude --log-file=G:\tmp\log\rsync_REMOTE_backup.log G:/DESKTOP/Automotive V:/public/documents

...but having kicked off this very run manually after much much longer than it normally runs I eventually killed it with CTRL-C in the CLI window, which produced the following error message:

Code: [Select]
...
2025/05/06 13:57:20 [126] rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
2025/05/06 13:57:20 [126] rsync: connection unexpectedly closed (78453 bytes received so far) [sender]
2025/05/06 13:57:20 [126] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9.1]
...

Soooo...I'm going to retry this again, but quite honestly I'm at a complete loss as to what's going on here.

This example here normally runs for all of two minutes...yup, that's all. Meanwhile today's run was 4 hrs. long and I suspect there was some kind of networking hangup that was encountered given that last error message.

This CTRL-C kill did not generate a process dump, which I think is the correct outcome.

EDIT #1
======

One more thing: take a look at the attached Theseus screenshot...RSYNC actually seems to create THREE processes, of which two have 2 threads, and the last one has a single thread...is that normal???


EDIT #2
======

...and yet the very next run of RSYNC, albeit with a different SOURCE & TARGET combo, completes just fine!!!

WHAT DA????
« Last Edit: May 06, 2025, 11:34:15 pm by Dariusz Piatkowski »

Dave Yeo

  • Hero Member
  • *****
  • Posts: 5400
  • Karma: +128/-1
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #10 on: May 07, 2025, 04:42:22 am »
Rsync forks into 3 processes here, backing up over USB.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1436
  • Karma: +28/-0
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #11 on: May 07, 2025, 04:59:09 pm »
Rsync forks into 3 processes here, backing up over USB.
Ok, cool...thanks for confirming this Dave...i.e. nothing unusual/suspect going on here (at least as far as RSYNC goes  :-\)

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1436
  • Karma: +28/-0
    • View Profile
Re: Troubleshooting process trap/dump...how?
« Reply #12 on: May 07, 2025, 05:30:10 pm »
UPDATE
======


OK, so perhaps not the real RC for the re-boot issue - I'm still working through that - but I do know what was causing the RSYNC issues I recently started to see.

In short: I have been experiencing problems with Lucide and opening PDFs that are stored on my NAS box.

Opening a single PDF is fine, 90% of the time I can open the 2nd one, 75% of the time I can open the 3rd one, but eventually Lucide comes up and no content shows and Lucide hangs. I can normally use the WPS TaskList to kill that Lucide instance, all others previously opened stay working, re-open the last doc and for the most part that's OK.

This only happens when I open multiple PDFs, which normally isn't the case, so for the most part it's not a big deal...but you know: when it becomes a nuisance is when you do in fact want multiple PDFs open, well, it was time to dig into this.

Seemingly the older versions of the Samba client & NetDrive plugins didn't show this, or at least I do not recall ever having these problems...so off I went redeploying older Samba Client & NetDrive combinations.

The combo I deployed most recently was samba-client-4.10.10-0.oc00.i686.rpm & ndpsmb-3.6.0-20190420.wpi, which means I was running Samba 4.10.10 client and NetDrive 3.6.0 plugin. (client stuff going into \usr\local\bin & \lib locations, etc.)

That did in fact improve my Lucide situation, considerably so, although not entirely.

HOWEVER

...what I did not know is that this combination (or most likely the Samba client) did mess around with the file DATES!!!

Sooo....the local & NAS dates would no longer match, as in: neither the CREATE nor MODIFIED dates were matching, and I couldn't determine what the pattern was either.

This in turn explained the strange RSYNC LOG results I would see where the most recent run of RSYNC if immediately re-ran would right away cause massive discrepancies (given my RYSNC options to match on MODIFIED dates and correct them).

I have now moved to samba-client-4.11.5-0.oc00.i686.rpm & ndpsmb-3.7.3-samba411-20220108.zip, which does correctly show the DATES...and that in turn has allowed RSYNC to breeze through it's regular runs because the vast majority of the files do NOT change.

So my RSYNC issue is "closed" now...lol

Appreciate all the feedback everyone!