Trying to solve server crash

[Ben]

My server has recently been having stability issues. It locks solid and leaves little indication as to what is going on.
In trying to track the problem down I noticed the following error repeated in my POPUP.LOG:

Code: [Select]

11-11-2019  23:23:20  SYS3175  PID 0012  TID 0001  Slot 0016
C:\OS2\SYSTEM\LOGDAEM.EXE
c0000005
15b112c9
P1=00000001  P2=001a2000  P3=XXXXXXXX  P4=XXXXXXXX
EAX=0005574c  EBX=ffffffe8  ECX=3ffff801  EDX=00242058
ESI=001a2000  EDI=0024403c
DS=0053  DSACC=d0f3  DSLIM=5fffffff
ES=0053  ESACC=d0f3  ESLIM=5fffffff
FS=150b  FSACC=00f3  FSLIM=00000030
GS=0000  GSACC=****  GSLIM=********
CS:EIP=005b:15b112c9  CSACC=d0df  CSLIM=5fffffff
SS:ESP=0053:00055554  SSACC=d0f3  SSLIM=5fffffff
EBP=00242058  FLG=00010206

ERRLOG.DLL 0001:000012c9

It has been a while since I looked at this stuff; any ideas?

[ivan]

Have you tested the server power supply voltages - if they fluctuate more than a few % the PS could need replacing.

The other thing is to run the full memtest tests from a bootable CD because bad memory is the biggest cause of instability and crashes.

[Pete]

Hi Ben

Unlike ivan I find the main cause of instability to be power supply or cooling related - I have yet to have a system fall over due to bad ram.

I suggest a visual examination of power supply and cooling system, clean or replace as necessary.

Also worth unplugging all leads inside the case - 1 at a time - and reinserting which will make sure connections are clean.

If the problem still persists then it is time to start checking ram and other bits of hardware...


All the above presumes there were no changes in software just before the server started showing problems.


Hope the above is of help.

Pete

[Fahrvenugen]

Also when you're looking inside the system, have a look at the capacitors on the motherboard.  If any are showing signs of bulging or leaking, this is a good indication for the cause of a crash.  If this is the case then spend some money on new capacitors and replace them (or find someone who's good with a soldering iron and see if they'll replace them for you).

[Andreas Schnellbacher]

have a look at the capacitors on the motherboard.

That's a good point. But in most cases and when the manufacturer has used capacitors of high quality, you won't see any deformation. You just can measure it. When capacity has decreased significantly, then it's time for soldering or ordering a new mainboard.

[Ben]

Thanks people.
I'll give those suggestions a try over the weekend as that will take a while.

Does anyone have any idea why that entry is repeated in the POPUP.LOG?

[Andi B.]

In contrary to the other speculations here in this thread I would guess if the problem is repentantly in Logdaem and errlog.dll then it's most probably not power supply related (includes capacitors).

Unfortunately I've no clue what logdamn and errlog do. No clue how to test if those program works as designed. I would check if errlog.dll is accessible and is the correct version and has the same CRC as a known working one from another system. Try to find someone who knows how to setup and test logdaem.

Of course all sort of problems are possible - failing harddisk, power supply, wrong LIBPATH, .... Only you know how old your system is and if you've used the cheapest parts or ones with better quality. F.i. if you're running a 2 years old Lenovo or HP standard system probability of failing power supply (capacitors) is much lower as when you're running a 15 years old server build up from the cheapest components available at ebay or alibaba.

For me until now there is far too less information in this thread to start speculating about HW issues.

[Ben]

At this point I'm trying to stick with what I know. I know that there is something popping up in the POPUP.LOG that shouldn't be there.
I can't say for certain if it is related to the crashes, but the frequency of it gives me pause for cause and it is the first place that I should look.

Either way, that issue has to be addressed. Once it is I will fix it and if my server instability perisists then I shall proceed onto the next most likely cause.

I have not seen that entry in the log before after using OS/2 for decades. It is also interesting to note that those entries started appearing on the 6th of November which is around the time that the server became unstable.

[Ian B Manners]

Hi Ben,

The only reason I can think of why the log daemon would have a problem is if the log file cannot be written to, assuming there are no problems with your harddisks. It is close to 20 years since I've had OS/2 log or error daemons enabled in OS/2 (apart from syslog) so have no memory of where the logs maybe kept.
Do a search of your boot drive and maybe consider backing any log files up if you need then then delete them.

[Fahrvenugen]

Only you know how old your system is and if you've used the cheapest parts or ones with better quality. F.i. if you're running a 2 years old Lenovo or HP standard system probability of failing power supply (capacitors) is much lower as when you're running a 15 years old server build up from the cheapest components available at ebay or alibaba.

Hi,

I'm not trying to cause a big debate here, but just about 2 months ago I had to replace a bunch of failing capacitors on a 4 year old HP enterprise-grade sever.  I also just took some 5 year old HP desktop machines out of service which have capacitors showing signs of failure.

Poor quality capacitors are still a problem.

[Ben]

There is always the possibility that this is a hardware issue, however, as long as the entry in the log is consistently the same program, so that has to be eliminated first. If it were never the same thing or no logs had any recent errorsin them then I would focus more on the hardware.

It is a hard lock though, not a soft lock; everything locks solid without indication as to what is causing it not even the mouse or any monitor moves. Not even the 3-finger salut does anything. And it is completely inconsistant as to when it will happen, for example the server has been up for over seven hours right now without incident. And then it might lockup solid every few minutes. I have replaced some hardware, (video card), and shuffled around where my USB mouse connector is, but nothing had any effect; nothing made it worse or better.

So, back to the issue of the logdaem entry in the POPUP.LOG. What does that program even do? I looked for some information on it, but there is very little within OS/2 itself on it, nor in any book that I have. The obvious answer is that it controls the logging of certain, (or all), log files that OS/2 generates, but that's just a guess. It may not be related to OS/2 itself, but something else running under the OS.

An easy test is to remove it from the CONFIG.SYS and see if anything different manifest. I think I'll give that a go.

[RickCHodgin]

Hi,

I'm not trying to cause a big debate here, but just about 2 months ago I had to replace a bunch of failing capacitors on a 4 year old HP enterprise-grade sever.  I also just took some 5 year old HP desktop machines out of service which have capacitors showing signs of failure.

Poor quality capacitors are still a problem.

I'm surprised to hear that about HP.  I've purposefully chosen to buy HP equipment because the machines I've gotten from them have been rock solid stable.  I've had one laptop I used daily for 5+ years that my wife is still using daily.  It's almost 9 years old now.  I have another HP desktop machine that was purchased in 2007 that I use for development.  It is absolutely stable, and because of its design I was able to increase the memory to 8 GB and CPU to quad-core 2.4 GHz, and still have quite a decent performer out of it 12+ years later.

I would buy HP again in a heartbeat.  Never tried an enterprise-grade server, however. :-)

[roberto]

I do not know if it could or not have something to do with your problem, but I in an HP laptop, very good team by the way. It happened to me that it was very unistable and finally removing the acpidaemon.exe works better than ever. You can kill this process and start it with the computer running and see differences.

[Ben]

This is not a new server setup, but a very long running one.

Nothing has really changed. I did install the latest XFolder a while back, but this latest problem has come in only recently... in the past week or so I shall say. Unless something can take a while to ferment, (a month or so), I cannot see any connection between the two. However, I probably should remove it and see if it makes any difference.

[Fahrvenugen]

Ben,

If it has been running for a while with XFolder without problems, then it is unlikely to be an issue there.  Over the years with OS/2 I've generally found that OS and software problems normally turn up fairly quickly, they don't normally just show up after a long time of stability.  But if that was the last major change made then certainly remove it and see if the problem fixes itself.

One thing to check though - make sure you've got free space on your drives, especially the one that the swap file is on.

Also - you mentioned this is running as a server.  Check log file sizes on any server software that you're running.  I recall once I had a situation where a MySQL server started crashing - eventually I figured out that the log that MySQL kept had become too big.  After deleting the log file (and I think I tweaked the configuration file to limit the log size or something) the stability came back.

You could also try running checkini on the system - just in case it is something wrong with the ini files.  Before you do this make a backup of your files just in case  something goes wrong.

And as previously mentioned, make sure that logdaem.exe and errlog.dll are not corrupted.

Just checking a few things, logdaem.exe is part of the first failure support technology diagnostic stuff, you could try REMing it out of your config.sys and see if the problem goes away.  I admit that after some googling I'm not finding a lot on logdaem.exe, but there is one article  suggesting it isn't really needed for everyday use and to try REMing it out.

I still am not willing to rule out power supply and / or hardware issues.  FFST could be picking up a hardware problem and throwing the error with logdaem catching it for display/logging.  I've also found that a lot of times when a system is completely locked up (3 finger salute, mouse, keyboard, all don't work) and the problem just starts happening after years of stability, those are the situations where I've often found a hardware issue to explain it.

Not sure if any of this will help.