Author Topic: Trap 000e and Sockets  (Read 1337 times)

Ben

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 3
  • Posts: 209
  • Know Thyself
    • View Profile
Trap 000e and Sockets
« on: November 23, 2019, 03:33:35 pm »
As some of you probably have noticed I have been trying to deal with some server issues that have built up over the years as a result of it being left unattended, (due to personal tragedy), now that I have the time.

I have most of them wrapped up and only one is outstanding; sometimes my server will trap;
Trap 000e; Sockets.
It doesn't happen often, but contrary to that, it did so twice yesterday which probably means that it is increasing in frequency. But whether that is true or not, I have to deal with it.

The problem with traps generated by TCP/IP socket I thought had been permanently deal with by IBM with their last TCP/IP update. As a result of that one seldom reads anything about it anymore.

Has anyone here experienced such issues and if so, have any pointers on where to look?
My guess would be crappy NIC drivers, but I need to have more to go on before spending money on something that may not solve the problem.

Thank-you.


ivan

  • Hero Member
  • *****
  • Thank You
  • -Given: 2
  • -Receive: 35
  • Posts: 1026
    • View Profile
Re: Trap 000e and Sockets
« Reply #1 on: November 23, 2019, 06:15:45 pm »
Reading the Config tool Database I see that
Quote
TRAP 000E - PAGE FAULT: The referenced page is not present in memory, or the procedure referencing the page does not have enough privilege to access the page. Contact software support.

Memory problems perhaps?

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 11
  • -Receive: 149
  • Posts: 2168
    • View Profile
Re: Trap 000e and Sockets
« Reply #2 on: November 23, 2019, 06:32:31 pm »
It could be a software problem, bad pointer. But if it is increasing in frequency and nothing changed software related, it could be hardware. Without at least a picture of the trap screen or better yet, a system dump and someone knowledgeable looking at it, it is really hard to say.
Swapping network card might be a test. There's lots of other things that could be going.
I had a bad CPU once, a Cyrix i686 which overheated due to the CPU fan dying. Thing was likely running as if overclocked too. Main symptoms was Netscape 4.61crashing a lot and my volume not working unless I loaded and used the DOS driver for the card (a PAS). Swapped in a real Pentium one day out of curiosity and those symptoms vanished.
The one time I had a memory module go bad, loaded Thunderbird and the system died. Lucked out in that the first module I removed brought the system back to life and it ran for quite a while with 25% less memory.

Ian Manners

  • Global Moderator
  • Sr. Member
  • *****
  • Thank You
  • -Given: 37
  • -Receive: 26
  • Posts: 323
  • I am the computer, it is me.
    • View Profile
    • ComKal Networks Australia
Re: Trap 000e and Sockets
« Reply #3 on: November 23, 2019, 06:40:27 pm »
Hi Ben,

What does inetver return?

[C:\]inetver
Version numbers of TCP/IP protocol drivers:
   SOCKETS.SYS: 6.3100
   AFOS2.SYS:   6.3100
   AFINET.SYS:  6.3100
[C:\]

And are you running the patched versions for SMP (tcpip32-safe.zip dated 16th August 2011) ?

"A patch intended to make tcpip32.dll multi-thread and SMP safe. that should prevent the crashes many people have when using Mozilla apps. Redirects calls made to unsafe routines and routes them to safe versions that were already present in the dll. Based on the tcpip32.dll included with eCS 2.x (2004-05-21  11:15  98,611). It may be compatible with earlier versions of this dll but that has not been tested. From Steve Levine."
Cheers
Ian B Manners

Ben

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 3
  • Posts: 209
  • Know Thyself
    • View Profile
Re: Trap 000e and Sockets
« Reply #4 on: November 23, 2019, 10:20:38 pm »
Memory?

Possibly. I have some identical modules here; I will swap them out and see.

Software
?

Yes.
I have been updating a few things mostly using ANPM. Other than that Paul's latest version of BIND; installing, (and subsequently uninstalling), a few versions of XWorkplace, (I have to leave it off the machine as it makes the server hard lock).

I going to have to do a search for duplicate variations of *.DLLs to see if there's a problem. However, long before any updates I was getting an occasional Trap 000e Sockets, maybe once or twice a year. Now, I have received three in the passed 16 hours; something is getting worse.
Though, other than the traps the server is now rock solid. So, I am getting there... nearly there.

Speaking of duplicate DLLs, didn't someone write a program specifically designed to find DLL dupes on the path?

With respect to capturing the 'trap screen' is there a program that can do that or is this a 'camera only' affair?



Hi Ben,

Hello again, Ian.

What does inetver return?

[C:\]inetver
Version numbers of TCP/IP protocol drivers:
   SOCKETS.SYS: 6.3100
   AFOS2.SYS:   6.3100
   AFINET.SYS:  6.3100
[C:\]

Inetver returns same.

And are you running the patched versions for SMP (tcpip32-safe.zip dated 16th August 2011) ?

Yes.
I found two identical copies in fact; one in C:\MPTN\ and another in C:\MPTN\DLL\.
Strange...
I shall delete the former and keep the latter.


"A patch intended to make tcpip32.dll multi-thread and SMP safe. that should prevent the crashes many people have when using Mozilla apps. Redirects calls made to unsafe routines and routes them to safe versions that were already present in the dll. Based on the tcpip32.dll included with eCS 2.x (2004-05-21  11:15  98,611). It may be compatible with earlier versions of this dll but that has not been tested. From Steve Levine."

In the past at sometime I must have copied the patched version right out of that *.zip.

I fully expect that I'm going to have to, (at some point soon), replace both the NICs in the server as I have never been completely satisfied with their performance, (driver/NIC combination).

Presently I am using the Realtek Gigabit 8111/8168/8169 driver (MultiMac32).

While the subject is up does anyone know of a kickass driver/NIC combination for OS/2, (PCI-E slot)?

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 11
  • -Receive: 149
  • Posts: 2168
    • View Profile
Re: Trap 000e and Sockets
« Reply #5 on: November 24, 2019, 01:58:44 am »
You have to use a camera to capture the trap screen, png is best for sharing, unless you set up a dump partition and set it up to dump the memory.
Realtek cards do seem to have issues, depending on the exact card, not sure of a good replacement

Lars

  • Sr. Member
  • ****
  • Thank You
  • -Given: 0
  • -Receive: 43
  • Posts: 425
    • View Profile
Re: Trap 000e and Sockets
« Reply #6 on: November 24, 2019, 12:03:42 pm »
I think this is all an issue where the protocol drivers (either SOCKETSK.SYS or AFINETK.SYS) run out of memory.
I seem to remember that I added parameters to CNTRL.EXE and also to AFINETK.SYS to rectify this problem.
Effectively, you increase the memory pool for CNTRL.EXE and AFINETK.SYS respectively.
Search the internet of what parameters CNTRL.EXE and AFINETK.SYS understand. I cannot have a look as I am far away from home …

http://www.edm2.com/index.php/CNTRL.EXE
The AFINETK.SYS parameters are listed in some OS/2 tcpip IBM Redbook that I cannot find right now.

Might also be of interest, the amount of open sockets:
https://www.os2world.com/forum/index.php/topic,60.msg393.html#msg393
« Last Edit: November 24, 2019, 12:11:07 pm by Lars »

Roderick Klein

  • Sr. Member
  • ****
  • Thank You
  • -Given: 0
  • -Receive: 18
  • Posts: 316
    • View Profile
Re: Trap 000e and Sockets
« Reply #7 on: November 24, 2019, 11:03:00 pm »
As some of you probably have noticed I have been trying to deal with some server issues that have built up over the years as a result of it being left unattended, (due to personal tragedy), now that I have the time.

I have most of them wrapped up and only one is outstanding; sometimes my server will trap;
Trap 000e; Sockets.
It doesn't happen often, but contrary to that, it did so twice yesterday which probably means that it is increasing in frequency. But whether that is true or not, I have to deal with it.

The problem with traps generated by TCP/IP socket I thought had been permanently deal with by IBM with their last TCP/IP update. As a result of that one seldom reads anything about it anymore.

Has anyone here experienced such issues and if so, have any pointers on where to look?
My guess would be crappy NIC drivers, but I need to have more to go on before spending money on something that may not solve the problem.

Thank-you.



Having managed for years the Mensys webservers (I have not seen these boxes for close to 5 years) you can have TRAPS in the different protocol drivers for different reasons. Some of our servers where up for weeks, but...

1) Changing the settings on the network static with any driver switches is not recommended.

2) What type of software are you running ? Like what web browser are you running and what version of the software ?
Do you load PHP ? Do you load Wordpress website for example.

3) In inetcfg.ini make certain that
keepalive       300     7800    0       7800    KeepAlive (sec) is set to 300 and not 7800.

4) The exhaustation of TCP/IP stack resources can be contributed to many issue's which you slowy need to sort out.

A) Are you having DDOS attacks on your server ?
B) Are web crawlers (Search engines) drowning the server in requests.
C) Keep an eye on netstats -s it shows the open sockets.  They should close after some time so IP stack resources are given back to the system.

Also some software I have seen on OS/2 has the tendency to not release TCP/IP sockets. Again at a faster rate exhausting the system.
This certainly has nothing todo with bad NIC drivers. Its purely a matter of tuning your server and confuring the server software correctly.

Roderick Klein

Ben

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 3
  • Posts: 209
  • Know Thyself
    • View Profile
Re: Trap 000e and Sockets
« Reply #8 on: November 25, 2019, 02:25:39 am »
Having managed for years the Mensys webservers (I have not seen these boxes for close to 5 years) you can have TRAPS in the different protocol drivers for different reasons. Some of our servers where up for weeks, but...

1) Changing the settings on the network static with any driver switches is not recommended.

2) What type of software are you running ? Like what web browser are you running and what version of the software ?
Do you load PHP ? Do you load Wordpress website for example.

3) In inetcfg.ini make certain that
keepalive       300     7800    0       7800    KeepAlive (sec) is set to 300 and not 7800.

4) The exhaustation of TCP/IP stack resources can be contributed to many issue's which you slowy need to sort out.

A) Are you having DDOS attacks on your server ?
B) Are web crawlers (Search engines) drowning the server in requests.
C) Keep an eye on netstats -s it shows the open sockets.  They should close after some time so IP stack resources are given back to the system.

Also some software I have seen on OS/2 has the tendency to not release TCP/IP sockets. Again at a faster rate exhausting the system.
This certainly has nothing todo with bad NIC drivers. Its purely a matter of tuning your server and confuring the server software correctly.

Roderick Klein

Hello again, Roderick.

I posted a list of my running servers here somewhere, but I don't see it now; it must have been eaten in cyberspace so here is the list of everything that runs continuously and are loaded at boot;


- FTPd
- Changi
- IMAPd
- Weasel
- LSwitch
- PMUptime
- INIServ
- mySQLd
- HTTPd, (apache)
- NAMEd
- NMBd
- SMBd
- Daytime
- ClipView
- UniWatch
- XFile
- SysLOGd
- Pluto, (INJoyFW)
- Gateway, (INJoyFW)
- CV_SERVER_OS2
- CAD_POP
- PC, (Process Commander)
- SSESCTL
- IRCd, (Hybrid. Off and on, right now off)

The rest are all typical for any OS/2 setup so I won't list them.

It's running on a Dell Optiplex 980, (i7) with 2,(4?) Gigs of RAM.

This is a dedicated server setup; I do not use it actively. It sits in the corner with the monitor turned off running 24/7.

PHP is in use. I do use SMF for my forum(s), (5 at the moment), with TinyPortal on two and a CMS based on Joomla. Also, five hand-written sites.

I will modify my inetcfg.ini as you suggested, but keep in mind that this server setup has been running stable for decades, with adjustments and upgrades, (hardware and software), over the years. I did change the inetcfg.ini back in February, but I have no idea why; most likely some minor tweak of little to no importance.

Here is my
inetcfg.ini:

Code: [Select]
#INETCFG: CURRENT DEFAULT MINIMUM MAXIMUM
keepalive     60 7800    0   7800 KeepAlive (sec)
tcpswinsize 131072 32768 1024 246723 TCP SendWindow Size
tcprwinsize 131072 32768 1024 246723 TCP RecvWindow Size
udpswinsize 41600 9216 1024 246723 UDP SendWindow Size
udprwinsize 83200 41600 1024 246723 UDP RecvWindow Size
lingertime     60   120    0 65535 Linger Time (sec)
probecnt      8     8    1      8 Number Of KeepAlive Probes

# IPForwarding is handle by IPGATE in Setup.cmd
# ipforward      0     0    0      1 IP Forwarding flag ON/OFF

mtudiscover      1     1    0      1 TCP Path MTU Discovery ON/OFF
arpkillc   1200 1200 180 15300 ARP Complete TimeOut (sec)
arpkilli    180   180   60   1200 ARP InComplete TimeOut (sec)
lipcstswinsize 20480 10240 1024 246723 LIPC Stream SendWindow Size
lipcstrwinsize 83200 10240 1024 246723 LIPC Stream RecvWindow Size
lipcdgswinsize   8192 2048 1024 246723 LIPC Dgram SendWindow Size
lipcdgrwinsize 16384 4096 1024 246723 LIPC Dgram RecvWindow Size
synattack      1     0    0      1 SYN Attack Flag ON/OFF
syncookie      0     0    0      1 SYN Cookie Flag ON/OFF
firewall      0     0    0      1 FIREWALL ON/OFF
multidefrt      0     1    0      1 Multiple Default Routes ON/OFF
winscale      1     1    0      1 TCP Window Scale ON/OFF
timestmp      0     0    0      1 TCP TimeStamps ON/OFF
realslow     20    20    1    100 TCP TW Q Slow Timeout ticks
perfhttp      1     1    0      1 Fast Path HTTP ON/OFF
tcpttl     64    64    1    255 TCP  TTL (hops)
udpttl     64    64    1    255 UDP  TTL (hops)
icmpttl     64    64    1    255 ICMP TTL (hops)
fragttl     60    60    1    255 Fragment TTL (sec)
reusetw      1     1    0      1 Reuse Time Wait ON/OFF
msl     30    30    1 16384 TCP MaxSegLifeTime (sec)
cc      0     0    0      1 TTCP function ON/OFF
sack      0     0    0      1 Selective ACK ON/OFF









Roderick Klein

  • Sr. Member
  • ****
  • Thank You
  • -Given: 0
  • -Receive: 18
  • Posts: 316
    • View Profile
Re: Trap 000e and Sockets
« Reply #9 on: November 25, 2019, 02:11:39 pm »
While I wish it was the case the fact the box has been running 24/7 for so long is simply no warranty at all.
Remember your box is connected to the internet. And as technology moves forward so do the hacking attempts.
But also search engines have bots that scan servers.

A specific URL on your server in Joomla, for example, can be called everytime by a script kiddy can pretty quickly exhaust the server its resources,
a denial of service. PHP based CMS systems such as Joomla can be pretty CPU and memory consuming.

Other questions are what version of Apache and PHP are you running ?

Sorry to have so many questions but having such a server requires you to tweak some of the settings per deamon.
And keep track of the logs to find what is causing this. If your server would be connected to the internet and all of the services would not be running you would most likely not be having these issue's with socketsk.sys traps.

Finding the issue's can be time consuming.

Roderick

Roderick

Ben

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 3
  • Posts: 209
  • Know Thyself
    • View Profile
Re: Trap 000e and Sockets
« Reply #10 on: November 25, 2019, 03:08:52 pm »
While I wish it was the case the fact the box has been running 24/7 for so long is simply no warranty at all.
Remember your box is connected to the internet. And as technology moves forward so do the hacking attempts.
But also search engines have bots that scan servers.

A specific URL on your server in Joomla, for example, can be called everytime by a script kiddy can pretty quickly exhaust the server its resources,
a denial of service. PHP based CMS systems such as Joomla can be pretty CPU and memory consuming.

All too true, unfortunately. 8\


Other questions are what version of Apache and PHP are you running?

Apache/2 2.29

PHP v5.4.34


Sorry to have so many questions but having such a server requires you to tweak some of the settings per deamon.

Not a problem, Roderick. I appreciate that you are taking the time to assist.

Unfortunately, I am a victim of my own server success; it has run so well for so long that I have forgotten all the details of setup and configure that I had to have to implement and fine tune it all, simply from an extended lack of use.



And keep track of the logs to find what is causing this. If your server would be connected to the internet and all of the services would not be running you would most likely not be having these issue's with socketsk.sys traps.

Finding the issue's can be time consuming.

Roderick

Yes, however, I have no choice as i need this up and stable.

I will put off all further upgrades, modifications and changes until I can nail down the Sockets/Trap 000e problem since it is the most important.

I keep all logs.

No lockup(s) since I changed the Keepalive to 300, but I am also taking my server down during the night  until I get this issue settled.


Ben

  • Full Member
  • ***
  • Thank You
  • -Given: 0
  • -Receive: 3
  • Posts: 209
  • Know Thyself
    • View Profile
Re: Trap 000e and Sockets
« Reply #11 on: December 04, 2019, 05:15:16 pm »
Well, unfortunately the Traps persist.

I have captured a the trap screen if anyone can interpret it I would appreciate it.




Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 46
  • Posts: 1251
    • View Profile
Re: Trap 000e and Sockets
« Reply #12 on: December 04, 2019, 09:58:24 pm »
Quote
I have captured a the trap screen if anyone can interpret it I would appreciate it.

I don't know if it will help, but look for SOCKETSK.SYS, and try that. You probably also need AFINETK.SYS. I don't remember what supplied them (probably one of the latest network fix packs), or if there may be other parts required.

I also know that Arca Noae has patched them, to be more SMP friendly, but I don't know if the patched versions are available for anything except ArcaOS (probably not, Arca Noae is allowed to distribute patched IBM code, but only for ArcaOS)..

Roderick Klein

  • Sr. Member
  • ****
  • Thank You
  • -Given: 0
  • -Receive: 18
  • Posts: 316
    • View Profile
Re: Trap 000e and Sockets
« Reply #13 on: December 05, 2019, 08:32:52 am »
Good point Doug. I however checked both binaries in FC (socketsk.sys and sockets.sys). I think both have the module name "sockets".
So I am not certain if the TRAP screen makes it possible to see the difference which driver is loaded.
Ben your config.sys lists the driver socketsk.sys and afitnek.sys with a letter k before the .sys ?

Roderick

Roderick Klein

  • Sr. Member
  • ****
  • Thank You
  • -Given: 0
  • -Receive: 18
  • Posts: 316
    • View Profile
Re: Trap 000e and Sockets
« Reply #14 on: December 05, 2019, 08:37:46 am »
Ben,

TRAP screens are really the last resort if a webserver is unstable.

I did not hear back from you if you checked your logs ? Is there a certain Apache LOG files that grows very quickly ?
After the TRAP check the last Apache LOG file to see if a specific URL has been hit many times.


We still need to have a clear answer if you are not left with a large amount of open sockets that do not close ?(Netstat -s_
Keep an eye on netstat -m

times failed to find space               = 0 ?

What happens to mbufs obtained from page pool            = 192 ?

That should not keep running up forever.

I still think something is hitting your systems (one of the services). Exhausting the system its resources.

Roderick