OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Hardware => Topic started by: Ben on November 23, 2019, 03:33:35 pm

Title: Trap 000e and Sockets
Post by: Ben on November 23, 2019, 03:33:35 pm
As some of you probably have noticed I have been trying to deal with some server issues that have built up over the years as a result of it being left unattended, (due to personal tragedy), now that I have the time.

I have most of them wrapped up and only one is outstanding; sometimes my server will trap; Trap 000e; Sockets.
It doesn't happen often, but contrary to that, it did so twice yesterday which probably means that it is increasing in frequency. But whether that is true or not, I have to deal with it.

The problem with traps generated by TCP/IP socket I thought had been permanently deal with by IBM with their last TCP/IP update. As a result of that one seldom reads anything about it anymore.

Has anyone here experienced such issues and if so, have any pointers on where to look?
My guess would be crappy NIC drivers, but I need to have more to go on before spending money on something that may not solve the problem.

Thank-you.

Title: Re: Trap 000e and Sockets
Post by: ivan on November 23, 2019, 06:15:45 pm
Reading the Config tool Database I see that
Quote
TRAP 000E - PAGE FAULT: The referenced page is not present in memory, or the procedure referencing the page does not have enough privilege to access the page. Contact software support.

Memory problems perhaps?
Title: Re: Trap 000e and Sockets
Post by: Dave Yeo on November 23, 2019, 06:32:31 pm
It could be a software problem, bad pointer. But if it is increasing in frequency and nothing changed software related, it could be hardware. Without at least a picture of the trap screen or better yet, a system dump and someone knowledgeable looking at it, it is really hard to say.
Swapping network card might be a test. There's lots of other things that could be going.
I had a bad CPU once, a Cyrix i686 which overheated due to the CPU fan dying. Thing was likely running as if overclocked too. Main symptoms was Netscape 4.61crashing a lot and my volume not working unless I loaded and used the DOS driver for the card (a PAS). Swapped in a real Pentium one day out of curiosity and those symptoms vanished.
The one time I had a memory module go bad, loaded Thunderbird and the system died. Lucked out in that the first module I removed brought the system back to life and it ran for quite a while with 25% less memory.
Title: Re: Trap 000e and Sockets
Post by: Ian Manners on November 23, 2019, 06:40:27 pm
Hi Ben,

What does inetver return?

[C:\]inetver
Version numbers of TCP/IP protocol drivers:
   SOCKETS.SYS: 6.3100
   AFOS2.SYS:   6.3100
   AFINET.SYS:  6.3100
[C:\]

And are you running the patched versions for SMP (tcpip32-safe.zip dated 16th August 2011) ?

"A patch intended to make tcpip32.dll multi-thread and SMP safe. that should prevent the crashes many people have when using Mozilla apps. Redirects calls made to unsafe routines and routes them to safe versions that were already present in the dll. Based on the tcpip32.dll included with eCS 2.x (2004-05-21  11:15  98,611). It may be compatible with earlier versions of this dll but that has not been tested. From Steve Levine."
Title: Re: Trap 000e and Sockets
Post by: Ben on November 23, 2019, 10:20:38 pm
Memory?

Possibly. I have some identical modules here; I will swap them out and see.

Software
?

Yes.
I have been updating a few things mostly using ANPM. Other than that Paul's latest version of BIND; installing, (and subsequently uninstalling), a few versions of XWorkplace, (I have to leave it off the machine as it makes the server hard lock).

I going to have to do a search for duplicate variations of *.DLLs to see if there's a problem. However, long before any updates I was getting an occasional Trap 000e Sockets, maybe once or twice a year. Now, I have received three in the passed 16 hours; something is getting worse.
Though, other than the traps the server is now rock solid. So, I am getting there... nearly there.

Speaking of duplicate DLLs, didn't someone write a program specifically designed to find DLL dupes on the path?

With respect to capturing the 'trap screen' is there a program that can do that or is this a 'camera only' affair?


Hi Ben,

Hello again, Ian.

What does inetver return?

[C:\]inetver
Version numbers of TCP/IP protocol drivers:
   SOCKETS.SYS: 6.3100
   AFOS2.SYS:   6.3100
   AFINET.SYS:  6.3100
[C:\]

Inetver returns same.

And are you running the patched versions for SMP (tcpip32-safe.zip dated 16th August 2011) ?

Yes.
I found two identical copies in fact; one in C:\MPTN\ and another in C:\MPTN\DLL\.
Strange...
I shall delete the former and keep the latter.

"A patch intended to make tcpip32.dll multi-thread and SMP safe. that should prevent the crashes many people have when using Mozilla apps. Redirects calls made to unsafe routines and routes them to safe versions that were already present in the dll. Based on the tcpip32.dll included with eCS 2.x (2004-05-21  11:15  98,611). It may be compatible with earlier versions of this dll but that has not been tested. From Steve Levine."

In the past at sometime I must have copied the patched version right out of that *.zip.

I fully expect that I'm going to have to, (at some point soon), replace both the NICs in the server as I have never been completely satisfied with their performance, (driver/NIC combination).

Presently I am using the Realtek Gigabit 8111/8168/8169 driver (MultiMac32).

While the subject is up does anyone know of a kickass driver/NIC combination for OS/2, (PCI-E slot)?
Title: Re: Trap 000e and Sockets
Post by: Dave Yeo on November 24, 2019, 01:58:44 am
You have to use a camera to capture the trap screen, png is best for sharing, unless you set up a dump partition and set it up to dump the memory.
Realtek cards do seem to have issues, depending on the exact card, not sure of a good replacement
Title: Re: Trap 000e and Sockets
Post by: Lars on November 24, 2019, 12:03:42 pm
I think this is all an issue where the protocol drivers (either SOCKETSK.SYS or AFINETK.SYS) run out of memory.
I seem to remember that I added parameters to CNTRL.EXE and also to AFINETK.SYS to rectify this problem.
Effectively, you increase the memory pool for CNTRL.EXE and AFINETK.SYS respectively.
Search the internet of what parameters CNTRL.EXE and AFINETK.SYS understand. I cannot have a look as I am far away from home …

http://www.edm2.com/index.php/CNTRL.EXE
The AFINETK.SYS parameters are listed in some OS/2 tcpip IBM Redbook that I cannot find right now.

Might also be of interest, the amount of open sockets:
https://www.os2world.com/forum/index.php/topic,60.msg393.html#msg393
Title: Re: Trap 000e and Sockets
Post by: Roderick Klein on November 24, 2019, 11:03:00 pm
As some of you probably have noticed I have been trying to deal with some server issues that have built up over the years as a result of it being left unattended, (due to personal tragedy), now that I have the time.

I have most of them wrapped up and only one is outstanding; sometimes my server will trap; Trap 000e; Sockets.
It doesn't happen often, but contrary to that, it did so twice yesterday which probably means that it is increasing in frequency. But whether that is true or not, I have to deal with it.

The problem with traps generated by TCP/IP socket I thought had been permanently deal with by IBM with their last TCP/IP update. As a result of that one seldom reads anything about it anymore.

Has anyone here experienced such issues and if so, have any pointers on where to look?
My guess would be crappy NIC drivers, but I need to have more to go on before spending money on something that may not solve the problem.

Thank-you.



Having managed for years the Mensys webservers (I have not seen these boxes for close to 5 years) you can have TRAPS in the different protocol drivers for different reasons. Some of our servers where up for weeks, but...

1) Changing the settings on the network static with any driver switches is not recommended.

2) What type of software are you running ? Like what web browser are you running and what version of the software ?
Do you load PHP ? Do you load Wordpress website for example.

3) In inetcfg.ini make certain that
keepalive       300     7800    0       7800    KeepAlive (sec) is set to 300 and not 7800.

4) The exhaustation of TCP/IP stack resources can be contributed to many issue's which you slowy need to sort out.

A) Are you having DDOS attacks on your server ?
B) Are web crawlers (Search engines) drowning the server in requests.
C) Keep an eye on netstats -s it shows the open sockets.  They should close after some time so IP stack resources are given back to the system.

Also some software I have seen on OS/2 has the tendency to not release TCP/IP sockets. Again at a faster rate exhausting the system.
This certainly has nothing todo with bad NIC drivers. Its purely a matter of tuning your server and confuring the server software correctly.

Roderick Klein
Title: Re: Trap 000e and Sockets
Post by: Ben on November 25, 2019, 02:25:39 am
Having managed for years the Mensys webservers (I have not seen these boxes for close to 5 years) you can have TRAPS in the different protocol drivers for different reasons. Some of our servers where up for weeks, but...

1) Changing the settings on the network static with any driver switches is not recommended.

2) What type of software are you running ? Like what web browser are you running and what version of the software ?
Do you load PHP ? Do you load Wordpress website for example.

3) In inetcfg.ini make certain that
keepalive       300     7800    0       7800    KeepAlive (sec) is set to 300 and not 7800.

4) The exhaustation of TCP/IP stack resources can be contributed to many issue's which you slowy need to sort out.

A) Are you having DDOS attacks on your server ?
B) Are web crawlers (Search engines) drowning the server in requests.
C) Keep an eye on netstats -s it shows the open sockets.  They should close after some time so IP stack resources are given back to the system.

Also some software I have seen on OS/2 has the tendency to not release TCP/IP sockets. Again at a faster rate exhausting the system.
This certainly has nothing todo with bad NIC drivers. Its purely a matter of tuning your server and confuring the server software correctly.

Roderick Klein

Hello again, Roderick.

I posted a list of my running servers here somewhere, but I don't see it now; it must have been eaten in cyberspace so here is the list of everything that runs continuously and are loaded at boot;

- FTPd
- Changi
- IMAPd
- Weasel
- LSwitch
- PMUptime
- INIServ
- mySQLd
- HTTPd, (apache)
- NAMEd
- NMBd
- SMBd
- Daytime
- ClipView
- UniWatch
- XFile
- SysLOGd
- Pluto, (INJoyFW)
- Gateway, (INJoyFW)
- CV_SERVER_OS2
- CAD_POP
- PC, (Process Commander)
- SSESCTL
- IRCd, (Hybrid. Off and on, right now off)

The rest are all typical for any OS/2 setup so I won't list them.

It's running on a Dell Optiplex 980, (i7) with 2,(4?) Gigs of RAM.

This is a dedicated server setup; I do not use it actively. It sits in the corner with the monitor turned off running 24/7.

PHP is in use. I do use SMF for my forum(s), (5 at the moment), with TinyPortal on two and a CMS based on Joomla. Also, five hand-written sites.

I will modify my inetcfg.ini as you suggested, but keep in mind that this server setup has been running stable for decades, with adjustments and upgrades, (hardware and software), over the years. I did change the inetcfg.ini back in February, but I have no idea why; most likely some minor tweak of little to no importance.

Here is my inetcfg.ini:

Code: [Select]
#INETCFG: CURRENT DEFAULT MINIMUM MAXIMUM
keepalive     60 7800    0   7800 KeepAlive (sec)
tcpswinsize 131072 32768 1024 246723 TCP SendWindow Size
tcprwinsize 131072 32768 1024 246723 TCP RecvWindow Size
udpswinsize 41600 9216 1024 246723 UDP SendWindow Size
udprwinsize 83200 41600 1024 246723 UDP RecvWindow Size
lingertime     60   120    0 65535 Linger Time (sec)
probecnt      8     8    1      8 Number Of KeepAlive Probes

# IPForwarding is handle by IPGATE in Setup.cmd
# ipforward      0     0    0      1 IP Forwarding flag ON/OFF

mtudiscover      1     1    0      1 TCP Path MTU Discovery ON/OFF
arpkillc   1200 1200 180 15300 ARP Complete TimeOut (sec)
arpkilli    180   180   60   1200 ARP InComplete TimeOut (sec)
lipcstswinsize 20480 10240 1024 246723 LIPC Stream SendWindow Size
lipcstrwinsize 83200 10240 1024 246723 LIPC Stream RecvWindow Size
lipcdgswinsize   8192 2048 1024 246723 LIPC Dgram SendWindow Size
lipcdgrwinsize 16384 4096 1024 246723 LIPC Dgram RecvWindow Size
synattack      1     0    0      1 SYN Attack Flag ON/OFF
syncookie      0     0    0      1 SYN Cookie Flag ON/OFF
firewall      0     0    0      1 FIREWALL ON/OFF
multidefrt      0     1    0      1 Multiple Default Routes ON/OFF
winscale      1     1    0      1 TCP Window Scale ON/OFF
timestmp      0     0    0      1 TCP TimeStamps ON/OFF
realslow     20    20    1    100 TCP TW Q Slow Timeout ticks
perfhttp      1     1    0      1 Fast Path HTTP ON/OFF
tcpttl     64    64    1    255 TCP  TTL (hops)
udpttl     64    64    1    255 UDP  TTL (hops)
icmpttl     64    64    1    255 ICMP TTL (hops)
fragttl     60    60    1    255 Fragment TTL (sec)
reusetw      1     1    0      1 Reuse Time Wait ON/OFF
msl     30    30    1 16384 TCP MaxSegLifeTime (sec)
cc      0     0    0      1 TTCP function ON/OFF
sack      0     0    0      1 Selective ACK ON/OFF








Title: Re: Trap 000e and Sockets
Post by: Roderick Klein on November 25, 2019, 02:11:39 pm
While I wish it was the case the fact the box has been running 24/7 for so long is simply no warranty at all.
Remember your box is connected to the internet. And as technology moves forward so do the hacking attempts.
But also search engines have bots that scan servers.

A specific URL on your server in Joomla, for example, can be called everytime by a script kiddy can pretty quickly exhaust the server its resources,
a denial of service. PHP based CMS systems such as Joomla can be pretty CPU and memory consuming.

Other questions are what version of Apache and PHP are you running ?

Sorry to have so many questions but having such a server requires you to tweak some of the settings per deamon.
And keep track of the logs to find what is causing this. If your server would be connected to the internet and all of the services would not be running you would most likely not be having these issue's with socketsk.sys traps.

Finding the issue's can be time consuming.

Roderick

Roderick
Title: Re: Trap 000e and Sockets
Post by: Ben on November 25, 2019, 03:08:52 pm
While I wish it was the case the fact the box has been running 24/7 for so long is simply no warranty at all.
Remember your box is connected to the internet. And as technology moves forward so do the hacking attempts.
But also search engines have bots that scan servers.

A specific URL on your server in Joomla, for example, can be called everytime by a script kiddy can pretty quickly exhaust the server its resources,
a denial of service. PHP based CMS systems such as Joomla can be pretty CPU and memory consuming.

All too true, unfortunately. 8\


Other questions are what version of Apache and PHP are you running?

Apache/2 2.29

PHP v5.4.34


Sorry to have so many questions but having such a server requires you to tweak some of the settings per deamon.

Not a problem, Roderick. I appreciate that you are taking the time to assist.

Unfortunately, I am a victim of my own server success; it has run so well for so long that I have forgotten all the details of setup and configure that I had to have to implement and fine tune it all, simply from an extended lack of use.


And keep track of the logs to find what is causing this. If your server would be connected to the internet and all of the services would not be running you would most likely not be having these issue's with socketsk.sys traps.

Finding the issue's can be time consuming.

Roderick

Yes, however, I have no choice as i need this up and stable.

I will put off all further upgrades, modifications and changes until I can nail down the Sockets/Trap 000e problem since it is the most important.

I keep all logs.

No lockup(s) since I changed the Keepalive to 300, but I am also taking my server down during the night  until I get this issue settled.

Title: Re: Trap 000e and Sockets
Post by: Ben on December 04, 2019, 05:15:16 pm
Well, unfortunately the Traps persist.

I have captured a the trap screen if anyone can interpret it I would appreciate it.



Title: Re: Trap 000e and Sockets
Post by: Doug Bissett on December 04, 2019, 09:58:24 pm
Quote
I have captured a the trap screen if anyone can interpret it I would appreciate it.

I don't know if it will help, but look for SOCKETSK.SYS, and try that. You probably also need AFINETK.SYS. I don't remember what supplied them (probably one of the latest network fix packs), or if there may be other parts required.

I also know that Arca Noae has patched them, to be more SMP friendly, but I don't know if the patched versions are available for anything except ArcaOS (probably not, Arca Noae is allowed to distribute patched IBM code, but only for ArcaOS)..
Title: Re: Trap 000e and Sockets
Post by: Roderick Klein on December 05, 2019, 08:32:52 am
Good point Doug. I however checked both binaries in FC (socketsk.sys and sockets.sys). I think both have the module name "sockets".
So I am not certain if the TRAP screen makes it possible to see the difference which driver is loaded.
Ben your config.sys lists the driver socketsk.sys and afitnek.sys with a letter k before the .sys ?

Roderick
Title: Re: Trap 000e and Sockets
Post by: Roderick Klein on December 05, 2019, 08:37:46 am
Ben,

TRAP screens are really the last resort if a webserver is unstable.

I did not hear back from you if you checked your logs ? Is there a certain Apache LOG files that grows very quickly ?
After the TRAP check the last Apache LOG file to see if a specific URL has been hit many times.


We still need to have a clear answer if you are not left with a large amount of open sockets that do not close ?(Netstat -s_
Keep an eye on netstat -m

times failed to find space               = 0 ?

What happens to mbufs obtained from page pool            = 192 ?

That should not keep running up forever.

I still think something is hitting your systems (one of the services). Exhausting the system its resources.

Roderick
Title: Re: Trap 000e and Sockets
Post by: Ben on December 05, 2019, 03:19:58 pm
I have both of those files in my CONFIG.SYS both in C:\MPTN\PROTOCOL.

SOCKETSK.SYS   8-24-14      128,512
AFINETK.SYS      9-20-15      318,976

I rotate my apache logs about once a month. Right now the biggest is around 2 meg.

With respect to netstat -m right now after running 6 hours this is what it shows:

Code: [Select]
mbufs obtained from page pool            = 10304
free mbufs                               = 8484
data mbufs                               = 0
header mbufs                             = 1
socket structure mbufs                   = 75
socket name mbufs                        = 43
socket option mbufs                      = 0
protocol control block mbufs             = 100
routing table mbufs                      = 1585
fragment reassembly mbufs                = 0
clusters obtained from page pool         = 1135
free clusters                            = 486
times failed to find space               = 0
times waited for space                   = 0


I will monitor InJoyFW GUI to see if see if something or someone is hammering the system but usually that will automatically block any IP/Port that is doing so.
Title: Re: Trap 000e and Sockets
Post by: Doug Bissett on December 05, 2019, 04:54:22 pm
Good point Doug. I however checked both binaries in FC (socketsk.sys and sockets.sys). I think both have the module name "sockets".
So I am not certain if the TRAP screen makes it possible to see the difference which driver is loaded.
Ben your config.sys lists the driver socketsk.sys and afitnek.sys with a letter k before the .sys ?

Roderick

I haven't seen a SOCKETS(K) trap, for a couple of years. I do see the odd AFINETK (with the K) trap, which is why I made the suggestion. Apparently, the AFINETK trap has something to do with SMP.

I will also mention, that ArcaOS includes a program SOCKTIDY.EXE, that is supposed to keep dead sockets from accumulating. Again, I am not sure if that is available outside of ArcaOS. Apparently, it helps with servers, but I find that it is not good for normal use. Currently, it is started in C:\TCPIP\bin\B4TCP.CMD but it was started in a different place in earlier versions of ArcaOS.
Title: Re: Trap 000e and Sockets
Post by: Lars on December 05, 2019, 11:51:11 pm
Try this:
DEVICE=C:\MPTN\PROTOCOL\SOCKETSK.SYS /mbuf:512 /mem:3600

You can try higher values for mbuf and mem.
If that stretches the time until next crash then SOCKETSK.SYS likely has a bug on allocating a buffer as the trap tells me that it is trying to copy memory from an address beyond the limit it can access.
I wouldn't know how to fix this error as no source code exists for SOCKETSK.SYS.
Title: Re: Trap 000e and Sockets
Post by: Ben on December 06, 2019, 02:46:37 am
OK, Lars.

I have made the modifications, now I shall see what happens.

Title: Re: Trap 000e and Sockets
Post by: Lars on December 06, 2019, 09:24:43 am
Forgot:
1) do not add drivers or modify config.sys ordering on this system. That will allow to compare a trap screen with the first trap screen you already posted.
2) take picture of trap screen if trap happens again.
Check yourself the value of CR2 on trap screen: if it is not higher than the first time,you should increase mbufs or mem
3) as an info: mbufs has relation to everything that says mbufs in netstat -m output and mem has relation to everything that says clusters in netstat -m output. Unfortunately I do not know the direct relation.
Title: Re: Trap 000e and Sockets
Post by: Ben on December 07, 2019, 02:20:57 pm
Thanks, Lars.

No issues yet, but it hasn't been long enough.

One thing of note is that some lines that were static before have changed... as expected netstat -m - posted for those who are following along;

Code: [Select]
mbufs obtained from page pool            = 4272
free mbufs                               = 2888
data mbufs                               = 0
header mbufs                             = 3
socket structure mbufs                   = 77
socket name mbufs                        = 43
socket option mbufs                      = 0
protocol control block mbufs             = 104
routing table mbufs                      = 1141
fragment reassembly mbufs                = 0
clusters obtained from page pool         = 1150
free clusters                            = 878
times failed to find space               = 0
times waited for space                   = 0
Title: Re: Trap 000e and Sockets
Post by: Lars on December 07, 2019, 03:58:22 pm
It's a good idea to monitor the output of netstat -m over time.
Maybe you can run netstat every 12 hours or so and dump the output to a file. Maybe that tells you something about free mbufs and free clusters.
Increasing these values in config.sys is just a hack to prevent the driver from ever asking for the last available buffer. Because I think that is when it fails.
Title: Re: Trap 000e and Sockets
Post by: Ben on December 09, 2019, 02:07:31 am
I have been doing that. Too bad there isn't any way to do that automatically and add the date/time.

Still no crash since adding those parameters though. 8)
Title: Re: Trap 000e and Sockets
Post by: Lars on December 09, 2019, 03:12:02 am
I think there is a tool called chron that allows you to run an application on a time schedule. I think it is installable via AN package manager ( it is hosted at Netlabs,ported by BWW).
It should also be possible to write a REXX script by calling a tool called sleep to put the script to rest. In the REXX script you can write to the log file whatever you want,including time and date.
Title: Re: Trap 000e and Sockets
Post by: Ben on December 10, 2019, 07:16:13 pm
I can do the scheduling through DragText, but writting the REXX is a hit and miss process for me.

I will take a look through Hobbes; maybe there is something there that I can adapt/modify.
Title: Re: Trap 000e and Sockets
Post by: Rene Hvidsoe on December 10, 2019, 07:48:22 pm
Hi Ben,

I have for some years ago used a schedule utility called  schedb2.zip it adds a schedule page to WPProgram objects.

I have used it for clamav update on hourly basis for some years. Easy to use.

Is found on hobbes:

http://hobbes.nmsu.edu/download/pub/os2/util/schedule/schedb2.zip (http://hobbes.nmsu.edu/download/pub/os2/util/schedule/schedb2.zip)

Rene Hvidsoe

Title: Re: Trap 000e and Sockets
Post by: Ben on December 10, 2019, 08:19:55 pm
Hello, Rene.

DragText does the same thing, adding a page to every WPS object, and it's one of the essential enhancers that I add upon installing OS/2.
One has to appreciate such a transparent, necessary application(?). 8)

But, I was unaware of schedb2; thanks for the heads up.
Title: Re: Trap 000e and Sockets
Post by: Lars on December 12, 2019, 08:38:41 pm
Code: [Select]
/* this is a REXX script */

signal on error name getout
signal on failure name getout
signal on notready name getout
signal on error name getout
signal on halt name getout

LogPath = VALUE('LOGFILES',,'OS2ENVIRONMENT')
LogFile = LogPath'\netstat.log'

'@del /F 'LogFile' 2>&1 >NUL'
rc = stream(LogFile,'C','OPEN WRITE')

'@rxqueue /CLEAR'
say 'Press any key to end ...'
do i=1 to 10
   '@netstat -m | rxqueue'
   string = '********************************************************* 'Date()' *** 'Time()' ***'
   rc = lineout(LogFile,string)
   do j=1 to queued()
      parse pull string
      rc = lineout(LogFile,string)
   end
   if chars() > 0 then do
      leave
   end
   '@sleep 3600s'
end

getout:
rc = stream(LogFile,'C','CLOSE')
return 0
Title: Re: Trap 000e and Sockets
Post by: Ben on December 15, 2019, 05:52:44 pm
Thanks for that Lars.

Just incase  you are wondering, I haven't had a Trap since I changed my CONFIG.SYS as you suggested.

Hopefully the problem has been solved. 8)

Now, I can move onto, (or back to), the other issues.
Title: Re: Trap 000e and Sockets
Post by: Lars on December 15, 2019, 07:39:38 pm
Let's hope it's going to stay that way.