OS2World OLD-STATIC-BACKUP Forum

OS/2 - Technical => Setup & Installation => Topic started by: RobertM on 2010.04.12, 21:45:07

Title: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.12, 21:45:07
Hello all,

I am looking for information on the PROCESSES= statement for the config.sys file in regards to WSeB and eComStation. The info I have found so far is very vague.

This is for a server that is generally under high load running Apache. Any help would be greatly appreciated.

Best,
Robert
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: chennecke on 2010.04.15, 19:53:23
Well, there simply is not much information available. All I ever found -- including that which Scott Garfinkle provided me with -- is in the XWorkplace documentation. Be careful with that setting since as far as I understood it, increasing the number of possible processes reduces the available shared memory.

That said, running high-traffic sites with Apache under OS/2 is pretty much a no-go, at least if also PHP and MySQL are involved. AFAIK the problem is that sockets/processes are opened/created but never properly closed/terminated. IIRC Apache 1.3.x and 2.x have slightly different problems. Increasing the number of processes could maybe postpone the point of failure but will not be able to actually solve the problem. You probably will have to reboot the machine regularly (which OS2.org does). What you CAN do is to reduce the risk of failures by using Squid to "shield" Apache so the problematic conditions will not occur.

Try contacting Roderick Klein for some advice. Mensys are probably those with the most experience in that area.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.15, 19:59:02
Yup... and the problem seems to be Apache or PHP... killing it (Apache) seems to free the tied up resources.

It turns out I think I can write a script that does that in under one second... so... will try that for now till I can get some ideas as to possible fixes.

Currently I am running into the problems you noted at very low loads (compared to our main LDGW box).
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.20, 16:59:01
Here's something interesting I have noted:

When this system bombs, it's a crash in LIBC063.DLL and/or PHP5.dll that seems to cause it (and the related out of resources) even with a bunch of memory free.

I get a SYS3171, with a crash in httpd.exe caused by one or both of those DLLs. MySQL at that point becomes unusable as well until I kill it and restart it.

Or, inotherwords, once httpd.exe crashes via one of those modules, MySQL needs to be restarted in order to free the resources that are tied up.

On another machine, running the same version of MySQL which is only accessed via REXX and the MySQL.EXE CLI app, I have never had to restart due to lack of resources (even though it handles as many transactions).


Now, the differences between the systems are as follows:


System that "crashes"System that doesnt crash
Uses ApacheUses LDGW
Uses PHPUses REXX
2GB RAM4GB RAM
1536 VirtualAddressLimitVAL not set
Combination HPFS386/JFS (32MB/64MB)Combination HPFS386/JFS (32MB/64MB)
1600 Threads1600 Threads
Single CPUSMP (Quad Core)
UNI KernelSMP Kernel
Roughly 30,000 MySQL transactionsUpwards of 100,000 MySQL transaction
Roughly same number of web requestsRoughly same number of web requests
MySQL 5.0.51MySQL 5.0.51
Apache/2.2.8LDGW 4.6.2.8
PHP/5.2.5NO PHP
Apache uses ??? threads and processes???LDGW creates 1000 threads and manages them itself
LDGW presumably handles it's own processes for handling REXX scripts
No REXX used by Apache.

Dunno if that is helpful in determining the location or cause of the resource issues...
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: chennecke on 2010.04.20, 17:43:09
I think you won't have much luck solving the issue here. IIRC there is an Apache for OS/2 mailing list. You could also try raising the issue on the netlabs.org IRC channel.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.20, 18:10:50
Quote from: chennecke on 2010.04.20, 17:43:09
I think you won't have much luck solving the issue here. IIRC there is an Apache for OS/2 mailing list. You could also try raising the issue on the netlabs.org IRC channel.

I'm thinking of doing so... but once I (or we - ie: others in the community with more Apache/Warp experience than I) track down whether it's Apache, libc063, or PHP - or how MySQL interacts with them - or the UNI setup that this OS/2 machine uses)...

The interesting things I found are: whatever is causing this cannot be alleviated without killing MySQL and restarting it - even though in a non-Apache/PHP world, MySQL doesnt run into these issues.

So... is it LIBC063? Is it Apache? Is it PHP? Is it processes or connections left open by PHP? Or ones left open by Apache via PHP? or is it that PHP and/or Apache are making extensive use of LIBC063? And why doesnt use of REXX and RxSQL cause a similar problem?

One thing I did not note on my earlier post is that both systems use CREXX (though of course, the Apache system isn't actually using REXX to facilitate any requests).


(thus, these are my questions to anyone more familiar with this than I - before I post the issues elsewhere - to help determine where the "elsewhere" is that I should be posting these to).

So... before I go posting there, I was hoping to gain some more insight to ensure the Apache list is where I should be posting (as opposed to looking for a newer or fixed version of LIBC063, or posting in the PHP related groups, or so on).

Or perhaps there is simply something that gets handled differently with an SMP version of OS/2? In which case, it would seem that the UNI versions are experiencing the problems and not the SMP versions of OS/2? That of course may mean it's an OS/2 issue related to how the resources are handled on UNI versions.

Sadly, I dont have an Apache/PHP/MySQL setup on an SMP version of OS/2 yet... does anyone who has such a setup run into this problem even though it's on an SMP version of OS/2 on an SMP computer? If so, is this one of the issues that gets alleviated by running the SMP kernel and other code on a non SMP machine?

Best,
Robert
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.04.21, 05:27:24
Hi,

I had been running into exactly the same issue a while back, and there was a lot of discussion on the Apache/2 email list about the problem (others had it too).

I never was able to find a reliable solution.  My setup was Apache2, PHP, and MySQL.  I ran this on both SMP and non-SMP machines and kept running into the same issue - the whole thing would eventually (after a day or so) get excessively slow and then lock up (usually requiring a reboot).  As for specific versions, I tried multiple different versions of Apache, PHP, and MySQL (somewhere I have notes on all the combos I tried).  I never did find one to be more reliable then another.

As much as I hated to do so, I needed to get that server up and running, so I ended up going with another OS for that server.

Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Ben on 2010.04.21, 14:34:28
Quote from: RobertM on 2010.04.20, 18:10:50

Or perhaps there is simply something that gets handled differently with an SMP version of OS/2? In which case, it would seem that the UNI versions are experiencing the problems and not the SMP versions of OS/2? That of course may mean it's an OS/2 issue related to how the resources are handled on UNI versions.

Sadly, I dont have an Apache/PHP/MySQL setup on an SMP version of OS/2 yet... does anyone who has such a setup run into this problem even though it's on an SMP version of OS/2 on an SMP computer? If so, is this one of the issues that gets alleviated by running the SMP kernel and other code on a non SMP machine?

I do not know if any of this will help you Robert, but I had similar problems with Apache/2 2.*, mySQL and PHP a year or two ago; I had upgraded from older versions.

At that point several others on the OS/2 ISP mailing list were having similar problems.

I can't tell you exactly what fixed the problem here, (which had become a major nuisance), but I'll tell you what I did and maybe it will give you a few more clues.


I switched over to the SMP kernel where I was only using one core in a dual core system before.

I backed up my conf files and SQL database, deleted and re-installed A/P/mS using the then latest version of AMP, (there have since been two or more releases since).

Though that's not very specific, I hope that some of this helps you.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.04.21, 16:57:57
Ben,

If you have a stable system, could you possibly check to see which versions / builds of Apache, PHP and MySQL you're running?

Also, do you happen to know which build / version of the SMP kernel you're running? 

I had never considered that it might be a kernel issue, but all the machines I had the problem with were running 14.104a (one on SMP with 2 processors, the other with the Uni kernel and 1 processor).

I'd love to try switching that system back over to OS/2 (in fact, I have 2 other OS/2 servers running at the site where I had the issue that I'd initially intended to run that website on, only to give up and set up a 3rd computer - just to run a Wordpress website - because I couldn't have my other 2 OS/2 servers - which normally the only downtime they have is when I reboot them for some reason or another - locking up on me on a daily basis).

Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.21, 18:28:58
Quote from: Fahrvenugen on 2010.04.21, 05:27:24
Hi,

I had been running into exactly the same issue a while back, and there was a lot of discussion on the Apache/2 email list about the problem (others had it too).

I never was able to find a reliable solution.  My setup was Apache2, PHP, and MySQL.  I ran this on both SMP and non-SMP machines and kept running into the same issue - the whole thing would eventually (after a day or so) get excessively slow and then lock up (usually requiring a reboot).  As for specific versions, I tried multiple different versions of Apache, PHP, and MySQL (somewhere I have notes on all the combos I tried).  I never did find one to be more reliable then another.

As much as I hated to do so, I needed to get that server up and running, so I ended up going with another OS for that server.



I've only run into one crash so far, and that crash was in JFS (which could be because when I got this system back up and running, I neglected to upgrade the JFS drivers to the latest versions - never had a JFS crash on my SMP JFS box).

I am suspecting that if the need arised for starting other processes from core subsystems, a crash is possible though - perhaps that is what happened with JFS on this box.

I have tried the following via a REXX script to try to resolve this (which is part of why I am sure it's not Apache itself - or rather not solely Apache).

What I have noted is that memory that Apache (and PHP?) had allocated does indeed get freed. But, still in a day or two (seems dependent on server usage) I still run into the same resource problem - which first results in slower connections, no responses to connections and then a crash in PHP and/or LIBC063 DLLs taking Apache out with it. Occassionally, the crash is in MySQL (but I'd say that's a one out of ten crashes scenario).

If I catch the box during the "inbetween time" I can kill Apache and MySQL and restart them - and everything is once again fine for about the same period of time. At this point, killing and restarting Apache itself is not sufficient to resolve the issue.

Running at 500+ requests per second, the LDGW/REXX/MySQL box does not run into those issues and happily chugs along... I've pushed that box to over 700 requests per second (at which time I run out of bandwidth - which eventually causes me to run out of TCPIP buffer space as WSeB tries buffering all the requests it cannot send upstream on the choked upstream pipe - but that's an obvious problem caused by not enough bandwidth and doesnt happen until we are at about 120% of our promised max).

Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.21, 18:31:44
Quote from: Ben on 2010.04.21, 14:34:28

Though that's not very specific, I hope that some of this helps you.[/color]

Hi Ben,

That may indeed be helpful. If you could check on one other thing for me... what version (or filesize or other version indicator) is the LIBC063 DLL on your system?

Since MySQL seems part and parcel to my problem on the Apache system - yet exhibits no problems on the SMP system... I think (assuming it's not the version of LIBC063 on the Apache system - or the way it acts on SMP vs UNI) that maybe you are on to something.

Any and all version numbers (Apache, PHP, MySQL, LIBC063 - or anything else you think relevant) would be greatly appreciated.

Best,
Robert
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: abwillis on 2010.04.21, 19:21:44
Just over a week before this thread was started I had received a reply from Lewis Rosenthal that indicated this is certainly not an uncommon problem.  http://www.warpstock.org was essentially down and Lewis said it was probably Apache out of resources and that he was having to restart his own servers about every 5 days to keep the support calls from coming in that email was crawling. 
In the back of my mind I keep thinking there was a discussion about this several years ago (maybe about the time that Apache 2.0 was released) but I can't find anything nor do I recall if any type of solution other than rebooting the system was found.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.04.21, 20:31:33
I will add to this, I think it is something to do with either PHP and / or MySQL, and how they interact with Apache.

For years I have been using Dink's Web/2 with the CGI PHP and MySQL, and have never run into problems.

I have found that if I'm serving static web pages using Apache2 doesn't seem to cause any problems, Apache can run for days, weeks, etc.

I've even found that when Apache is using PHP. but*not* trying to access MySQL it can run fine without problems (using the Apache PHP DLL module)

But when I put it all together where it is trying to access a MySQL database via PHP served through Apache, that's when I ran into problems.

What's even more interesting, when the system seems locked up, Apache will still occasionally be able to serve up static HTML files, which suggests that Apache isn't entirely frozen, the lockup is somewhere in PHP or MySQL.

I had thought potentially a MySQL - PHP issue, however MySQL - PHP runs fine when used with Dink's Web/2 / PHP / MySQL

As for a potential issue with the Apache PHP DLL, I tried running it under Apache with the PHP CGI (.EXE) - the same one / setup that I've never had issues with under Web/2, and I ran into the same issue under Apache.

So where's the problem?  I never found it and gave up looking.

Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.22, 03:57:46
So... this time, MySQL crashes - but POPUPLOG.OS2 shows Apache crashed in LIBC063 - followed immediately by Apache crashing again (weird, huh?) in PHP5.DLL

Even more interesting is that Apache was still running - while MySQL was not (nor was there an entry in POPUPLOG for MySQL).

Now, I am even more baffled and beginning to understand why this is so difficult to track down.




Should I start looking for a shared, conflicting DLL? Does one have some LIBC063 code compiled into it, or it's own version of the DLL someplace?
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.04.22, 05:53:23
You can definitely try looking for DLL's, if you have multiple versions of the same DLL on your system this might be part of it. 

I do wish we had a PHP module (or a way to make PHP work reliably) with LDGW, and the ability to add some of the Apache features (such as the URL re-writing stuff) to LDGW.  LDGW is a great web server, unfortunately it never seemed to get the recognition it deserved.

Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.22, 07:04:53
Quote from: Fahrvenugen on 2010.04.22, 05:53:23
You can definitely try looking for DLL's, if you have multiple versions of the same DLL on your system this might be part of it. 

I do wish we had a PHP module (or a way to make PHP work reliably) with LDGW, and the ability to add some of the Apache features (such as the URL re-writing stuff) to LDGW.  LDGW is a great web server, unfortunately it never seemed to get the recognition it deserved.



I wish I had the time to learn the GWAPI stuff... I've got a copy of VAC++ v3 just sitting here begging for such use.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Ben on 2010.04.22, 15:42:12
Quote from: RobertM on 2010.04.21, 18:31:44
Hi Ben,

That may indeed be helpful. If you could check on one other thing for me... what version (or filesize or other version indicator) is the LIBC063 DLL on your system?

Since MySQL seems part and parcel to my problem on the Apache system - yet exhibits no problems on the SMP system... I think (assuming it's not the version of LIBC063 on the Apache system - or the way it acts on SMP vs UNI) that maybe you are on to something.

Any and all version numbers (Apache, PHP, MySQL, LIBC063 - or anything else you think relevant) would be greatly appreciated.

Server Version: Apache/2.2.15 (OS/2) PHP/5.2.13
Server Built: Mar 7 2010 09:01:34

DB Type: mysql
DB Version: 5.1.36

LIBC063:      1,349,060   6/11/2007

SMPKernel:    908,650    6/15/2007

ACPI:               247,182  10/02/2008

I hope that helps.

Any more questions feel free to ask.

If I think of anything additional I'll post that too.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.22, 21:58:29
Thanks Ben...

Seems I have some upgrading of PHP and MySQL and Apache to do... LIBC063 is same size, and no other versions on the system.

In the meantime, I think I will try killing MySQL (using MySQLADMIN shutdown) at the same time I kill and restart Apache. The key again is to ensure that both restart in different processes to free memory and other resources. I've got it currently set at a 6 hour restart loop.

Of course, the bad thing is that means all caches are purged... but I can live with that if it "resolves" things until I do the updates and test that.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.24, 18:43:17
Well, so far this is the longest I have went without Apache or MySQL crashing - so the workaround seems to be working. Now... on to some upgrading to see what happens.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.04.24, 20:02:17
So just to confirm if I understand correctly, you've got a REXX script that:

1.  Kills MySQL using Mysqladmin shutdown
2.  Kills Apache with the apache shutdown script (which just kills the PID
3.  Restart Apache
4.  Restart MySQLD

Or are you restarting Apache and MySQL in the swapped order?

I'm guessing this is all in a REXX script called from one of the many schedulers out there.

I'm asking, I'd love to try this on the troublesome website that I have (the one I had to move off of OS/2) to see if I can get it running reliably under Warp.  I could potentially eliminate one machine (a machine which - its only task right now is to serve this troublesome wordpress website - overkill for a single machine, but it was the easiest way to get things up and running at the time...  as it is that machine requires a reboot every 2 - 3 weeks)
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: mobybrick on 2010.04.24, 21:32:04
Hi,

The problem is likely due to with problems in the low-memory arena. Many of the LIBC063 calls, in particularly PHP5, consume large amounts of low memory.

In one PHP5 app, I can exhaust OS/2 Apache2/PHP5/MySQL/LIBC063 in a single request. The request consumes 300Mb of low memory, and then exhausts with a weird error - sometimes a SYS3171 - but usually a LIC063 exception - but Thecus shows the shared memory arena as reaching 64Kb free and then PHP5 dies. On Win the calls run fine and consume 600Mb of RAM - but OS/2 can't allocate this and dies :( even on a 4Gb system.  As soon as exception occurs, the shared memory reverts to being free.

Under Win32 and Linux, PHP5 can routinely use 4-700Mb of RAM on busy systems - something we can't give it. Under OS/2, setting the PHP max memory limits aren't the problem as it is OS/2 that runs out memory.

Updating the PHP5/MySQL/Apache2 builds doesn't really help - remember, that even the order of things started in the startup.cmd can affect the amount of low memory available in the shared memory arena.

With some of the situations described above, it isn't how many concurrent threads or web sockets that are the problems per see, it is how much shared memory is being used - i.e. many clients can be supported if they use little memory, but few (or one) clients can be supported if that single request uses a lot of shared memory.

kLIBC, IMHO, needs re-working to make most of its calls use high-memory. Simply marking LIBC063.DLL to loadhigh doesn't working because all of the compiled apps that use the library need re-compiling AFAICS.

This needs fixing soon as soon the 300Mb of shared memory (its actually 512Mb, but 300Mb or so is all that is left once the system is booted to the desktop on most systems) is going to become a big problem.

Regards,
Moby.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.04.25, 00:03:39
Moby,

Thanks for the potential reasoning behind what is happening.

I guess my question is:  If it is a PHP / MySQL / Libc thing, then how come the same PHP based app can run fine when used on Dink's Web/2 using the PHP CGI exe, but fails / crashes on Apache2?

Of course, the main need for Apache is more advanced features (such as URL re-writing for SEO, etc) which Web/2 can't do.  But if it was purely a memory thing with PHP and MySQL grabbing huge amounts of low / shared memory, then I'd expect to see similar problems there.

Of course it is true that Web/2 doesn't use Libc063, so that might be part of it - just one more app (Apache) using Libc takes up too many resources.  I don't know.

Thanks
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.25, 01:31:14
I have some thoughts on this.

First, it's not just that LIBC063 exhausts the low memory arena on a single request (that isn't my problem). It also seems that (whether by design or by accident) under certain circumstances (for instance, when in use by Apache/MySQL/PHP) it does not clean up used memory.

That seems to be evidenced by the fact that under at least my setup, it's not a single large memory request causing the issue (that has never happened yet under my setup), but a collection of smaller ones over time that cumulatively exceed the available memory in the low memory arena. Otherwise, my "workaround" would not have alleviated my problem.

That's further evidenced by the fact that on my setup, no single request causes it. I can start a fresh copy of everything after a fresh boot and perform ONE certain medium sized (memory allocation wise) request with no problems. If the server is running half a day though, and I try the exact same single request with little to no activity on the server, things start to crash because of resource depletion.


So, perhaps the situation is some aspect of how Web/2 serves requests is different than the way Apache does. My guess is that Web/2 serves each in it's own process, and thus is freeing what memory is being used by LIBC063 upon the process's completion; while Apache perhaps is serving them all in one process via multiple threads, and thus not freeing the resources between requests until eventually the memory resources are depleted.


IF that theory is correct, that would mean that two things would need to happen... (1) LIBC063 would need to be recompiled to utilize the high memory calls implemented in WSeB and Warp 4 with the appropriate FPs, and (2) the more difficult part - something would need to be done to ensure that memory was freed at the completion of each call by a thread.

But again, that's based on my speculation on how Apache and Web/2 may be doing things differently than each other - which, not knowing enough about the underpinnings of either, I may be way off base.

Best,
Robert
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.25, 01:47:40
Quote from: Fahrvenugen on 2010.04.24, 20:02:17
So just to confirm if I understand correctly, you've got a REXX script that:

1.  Kills MySQL using Mysqladmin shutdown
2.  Kills Apache with the apache shutdown script (which just kills the PID
3.  Restart Apache
4.  Restart MySQLD

Or are you restarting Apache and MySQL in the swapped order?

The key being first, that I have selected a time limit that is under the amount of time it takes for them to crash on my system (in my case 6 hours works... I could probably get away with 10-12 hours, but due to search engine activity, that seems chancy).

Then I:

1. Have two REXX scripts:
--(a) One that starts MySQL in it's process
--(b) One that spawns Apache in a separate process from itself

2. The Apache script (since it is free as soon as it spawns Apache) waits 6 hours then (in a separate spawned process space) kills Apache (using the shutdown.cmd script, with some changes in the paths of the statements in it to ensure it finds what it needs)

3. The Apache Script then kills MySQL by calling MySQLAdmin shutdown in a separate process.

4. The MySQL script regains control once MySQL terminates. To ensure resources are freed, it spawns a new process and loads itself. The new process of that script then restarts MySQL (Step 1a)

5. The Apache script spawns a new process which loads the same Apache script which spawns another process where Apache starts (Step 1b)

6. The original scripts exit and close.

Not the neatest, and I can clean it up a little I am sure... but as long as my sleep value is small enough, it works as each original process is always killed, closed and resources freed (via killing and closing).

Quote from: Fahrvenugen on 2010.04.24, 20:02:17
I'm guessing this is all in a REXX script called from one of the many schedulers out there.

Yes it's all in a couple REXX scripts, and no, it doesnt use any of the schedulers out there... the ones I have found seem dependent on LIBC063 and thus dont solve the problem. It simply calls SysSleep(SLEEPTIME) and spawns new copies of itself.

Quote from: Fahrvenugen on 2010.04.24, 20:02:17
I'm asking, I'd love to try this on the troublesome website that I have (the one I had to move off of OS/2) to see if I can get it running reliably under Warp.  I could potentially eliminate one machine (a machine which - its only task right now is to serve this troublesome wordpress website - overkill for a single machine, but it was the easiest way to get things up and running at the time...  as it is that machine requires a reboot every 2 - 3 weeks)


I can send you the scripts if you want. They are very simple. And as I said, I am sure that they can be done better. They are not done better currently because originally, it was one script that killed and restarted Apache... and once I realized that did not solve the problem, I cobbled on the functionality to kill and restart MySQL and created the MySQL script. My priority at the time was to get the site working as quickly as possible - as opposed to creating a polished set of scripts.

Best,
Robert
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: mobybrick on 2010.04.25, 01:50:09
Quote from: RobertM on 2010.04.25, 01:31:14
First, it's not just that LIBC063 exhausts the low memory arena on a single request (that isn't my problem). It also seems that (whether by design or by accident) under certain circumstances (for instance, when in use by Apache/MySQL/PHP) it does not clean up used memory.

That seems to be evidenced by the fact that under at least my setup, it's not a single large memory request causing the issue (that has never happened yet under my setup), but a collection of smaller ones over time that cumulatively exceed the available memory in the low memory arena. Otherwise, my "workaround" would not have alleviated my problem.

That's further evidenced by the fact that on my setup, no single request causes it. I can start a fresh copy of everything after a fresh boot and perform ONE certain medium sized (memory allocation wise) request with no problems. If the server is running half a day though, and I try the exact same single request with little to no activity on the server, things start to crash because of resource depletion.

So, perhaps the situation is some aspect of how Web/2 serves requests is different than the way Apache does. My guess is that Web/2 serves each in it's own process, and thus is freeing what memory is being used by LIBC063 upon the process's completion; while Apache perhaps is serving them all in one process via multiple threads, and thus not freeing the resources between requests until eventually the memory resources are depleted.


Possibly, but I have an alternative theory on this. You will need to use Theucus to confirm it, however.

Firstly, Web/2 uses a single process for webserving - it's Apache2 that uses multiple processes. I suspect that Apache2 simply gives you another 50Mb of working space in the shared memory arena, so the system will last long.

What's happening over time is that I suspect that the bits of memory that Apache uses for working memory (remember, Apache spawns a new process, a 'fork' for each client) and this grows over time as clients are serviced. These will use more memory and this is fine. They might leak memory, but I have not seen it. What does happen though is that the shared memory arena becomes fragmented - OS/2 can't find enough *contiguous* memory in the shared arena for a large request. So your large request fails at the end of the day because the shared memory is a mess - used by a DLL, LIBC063, which never drops out memory but is constantly alloc/freeing memory - there might (or not) be a leak, but the biggest chunk of free memory becomes too small.

The fixes are this are twofold, IMHO

1. Building LIBC063 so that apps use high memory
2. Building apps with static linking under GCC - reducing reliance on a shared DLL in memory. Whether or not this is feasible I don't know - but it would appear to solve other problems IMHO.

We need to start  bounty to get some work done on kLibC as there have been no updates for a while. Getting some of the memory issues sorted, plus some of the bugs, and even maybe full support for pthreads would be very desirable.

Regards,
Moby
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.25, 02:55:58
Quote from: mobybrick on 2010.04.25, 01:50:09


Possibly, but I have an alternative theory on this. You will need to use Theucus to confirm it, however.

Firstly, Web/2 uses a single process for webserving - it's Apache2 that uses multiple processes. I suspect that Apache2 simply gives you another 50Mb of working space in the shared memory arena, so the system will last long.

...

Regards,
Moby

Ooops... I got the situation backwards. I thusly prefer your theory. Sadly, on one of my boxes (the Netfinity 7000 M10), Thesues will not run. I havent yet tried it on the other box, and cant afford to take that box down for any length of time.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.27, 21:16:02
So... it seems that Apache isnt always killed, but MySQL is... and that seems to alleviate problems.

Anyway, I am working on adding the use of one of the REXX memory monitoring DLLs, to allow restarting once the low memory arena starts to drop in memory.

Once I am done, I will post the scripts...
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.28, 23:28:42


The script:
- I need to kill Apache from within the running script (not enough resources to kill it by spawning a new process)
- I'm finding MySQL seems to be the bigger problem - once it is killed and restarted, a ton of resources seem to be freed.
- Digging for the REXX Extension DLL that reads available memory - I think it was RXU... I will find it and see what I can do with that.


So... soon I will have something up...
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: mobybrick on 2010.04.29, 00:54:54
Interesting... but the script will need to make sure that MySQL is shut down nicely or db corruption could occur (or transactions borked)... but I guess you already thought of this :)

Look forward to see what you get cooked up. Doesn't solve my problems tho where one or two big requests need 3-400Mb of memory - and OS/2 can never provide it...
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.29, 00:57:08
Quote from: mobybrick on 2010.04.29, 00:54:54
Interesting... but the script will need to make sure that MySQL is shut down nicely or db corruption could occur (or transactions borked)... but I guess you already thought of this :)

Yup. It uses MySQLAdmin shutdown to perform an orderly shutdown.


Quote from: mobybrick on 2010.04.29, 00:54:54
Look forward to see what you get cooked up. Doesn't solve my problems tho where one or two big requests need 3-400Mb of memory - and OS/2 can never provide it...

Nope sadly it doesnt. Seems we need a few things... libc063/Apache/MySQL/PHP recompiled to make use of the new high mem calls, and something that can make free memory contiguous.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.04.29, 02:42:45
I've had a few more ideas...

I can monitor MySQL's load and look for times it isn't in use before calling a shutdown (and if it is in use, and memory is running out and it still is in use while exceeding x number of seconds/minutes force a shutdown anyway).

And may be able to do that with Apache as well (not sure... gotta test some things).

I've got some testing with RXU to do...
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys stateme
Post by: RobertM on 2010.05.09, 20:56:53
Here's the simple and somewhat error prone version of the script. The comments and notes in it say it all - but it does generally serve it's purpose.

If and when I finish fine tuning it, I will post a new one.


/* Start and Control MySQL and Apache */


/*
    THIS SCRIPT IS IN MY C:\Scripts Directory
    You must modify the 3rd to last line (in the main section of this script) to the location you place it on your system


    Currently, this script does NO checking to see if it actually succeeds in trying to kill
    either Apache or MySQL. If it fails, it will still try to spawn new copies.
*/


    /* Define Daemon Locations */
        ApacheDir='D:\SERVERDAEMONS\APACHE2\'           /* DONT FORGET TRAILING SLASH */
        MySQLDir="D:\SERVERDAEMONS\MYSQL5\BIN\"         /* DONT FORGET TRAILING SLASH */
        ApacheLogsDir="D:\serverdaemons\apache2\logs\"  /* DONT FORGET TRAILING SLASH */
            /* This entry is where httpd.pid will be located */

    /* Define wait time before killing and restarting MySQL and Apache */
        xHours=3

    /* SLEEP for 3 seconds to allow killing of the script if need be */
        rtc=SysSleep(3)



    /* Detach and Run MySQL */
    /*
        REMOVE any parameters in the "start... MySQLD.EXE" line below that you do not use:
        I am simply too lazy to change my my.cnf file while I am testing what works on my system
    */

        'start /c /b /min 'MySQLDir'MYSQLD.EXE --max_seeks_for_key=50 --max_connections=500 --bulk_insert_buffer_size=64M --key_buffer_size=96M --table_cache=128 --sort_buffer_size=32M --read_buffer_size=8M'


    /* Detach and Run Apache */

        'start "Apache" /c /b /min D:\SERVERDAEMONS\APACHE2\STARTUP.CMD'


    /* Wait xHours until kill and restart of services */
        Say "Waiting "xHours" hours from: "Time()
        SleepTime=60*60*xHours

        rtc=SysSleep(SleepTime)


    /* Request proper shutdown of MySQL and Apache */
        'mysqladmin shutdown'
        rtc=Kill_Apache()

        /*
            BOTH of these are called WITHIN this REXX procedure and MUST be called from it
            - If you try to detach these, then if there is a problem (out of resources) the
              detached processes will NOT be started and the resources will not be freed.

              Inotherwords, nothing will happen at all except this will exit with two out of
              resources errors and the server daemons will not be restarted.

            - I have moved the Apache "Shutdown" script into this as a "function"
              You need to ensure that KILL is available in your path statement
              - Inotherwords, you need to add an entry to the config.sys path statement
                or ensure you have placed KILL.EXE in an already defined locations such as
                \OS2\BIN
        */


    /*
        Assumes both services have been killed.
        At this point, to ensure all resources are freed (including those used by this script)
        we will spawn another copy of this script which will wait 3 seconds (as defined above)
        before restarting the services in their own process spaces.

        This script will then exit 3 seconds later to free any resources it is using.

        While this script should not be using much in the line of resources, I have found that
        on ObjectREXX (especially the version shipped with eCS 1.2MR) every REXX script seems to
        leak resources the longer it runs.

        Thus, killing and restarting this script, instead of just looping forever, will alleviate
        those ObjectREXX issues.
    */


    /* Time to restart this script in a new session and then exit and close this session */
        'start /c /b /min C:\SCRIPTS\Run_ApacheMysql.cmd'
        /*
            You may wish to remove "/min" from the above to allow you to easily find and kill this
            script in the 3 second window it gives you in the event it gets stuck in a restart loop.
           
            While that shouldnt happen, it's just a thought.
           
            To properly kill things if a loop occurs or there is another problem, find and kill THIS
            script first, then properly shut down MySQL and kill Apache - then restart this script.
        */


        rtc=SysSleep(3)


Return 0            /* This script exits here */





Kill_Apache:
    pid = linein(ApacheLogsDir"httpd.pid")
    'kill.exe -TERM 'pid
Return 0
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys stateme
Post by: RobertM on 2010.05.09, 21:02:27
A final note to those of you running into single requests that are eating all available resources... by tweaking the PHP.ini, httpd.conf and my.cnf files, you should be able to avoid that to some extent.

While doing so may decrease performance of some queries, it will increase uptime by not causing anything to hang due to lack of resources.

The way the script is currently written, it should mean that there should be no more than xHours (see script - currently defined as 3 hours) worth of downtime at any given cycle (because at xHours, both MySQL and Apache will be shut down and restarted, which will free up any resources).
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.05.10, 16:20:38
Thanks!

While that doesn't solve the memory / lockup problems, it does likely provide a work-around.  I'll have to give it a try.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.05.10, 19:57:44
Quote from: Fahrvenugen on 2010.05.10, 16:20:38
Thanks!

While that doesn't solve the memory / lockup problems, it does likely provide a work-around.  I'll have to give it a try.


I'm looking into some of the other ideas others have posted... such as trying to force an unload of libc063 (which definitely seems to be the problems according to Apache's error logs), and trying to re-align memory use to create contiguous free memory blocks.

I'm also working on integrating the use of RXU to monitor free memory space to cause an "unscheduled" shutdown and restart of Apache and MySQL if free memory drops below a certain user defined threshold.

Also, in my script, there is one line that is in error (or needs to be custom set)


    'start "Apache" /c /b /min D:\SERVERDAEMONS\APACHE2\STARTUP.CMD'



It should have read:


    'start "Apache" /c /b /min 'ApacheDir'STARTUP.CMD'


Of course, manually changing it to the correct path would work as well, but changing it to use the pre-defined variable at the top of the script makes it a lot easier to maintain if one decides to move their Apache installation to another path/drive - just then requires updating the variable definition at the top of the script.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.05.10, 20:48:06
Quote from: mobybrick on 2010.04.25, 01:50:09
2. Building apps with static linking under GCC - reducing reliance on a shared DLL in memory. Whether or not this is feasible I don't know - but it would appear to solve other problems IMHO.

Regards,
Moby

Moby, do you think simply number 2 would alleviate these issues? I've gotten some info from RXU and am seeing the following:

After a few hours of running:
- Free Shared Memory: 44MB
- Free Memory in Private Arena that the process can allocate: 291MB

After shutting down services:
- Free Shared Memory: 245MB
- Free Memory in Private Arena that the process can allocate: 291MB

After restarting services:
- Free Shared Memory: 223MB
- Free Memory in Private Arena that the process can allocate: 291MB


The key being it seems that the private arena has a big chunk of available memory - currently more so than the shared arena starts with. If each (PHP, MySQL and Apache) had LIBC063 functionality statically linked, would that mean that EACH would have that 291MB available to it?

I am also suspecting that on most systems, the amount will be more... as I am allocating a bit of memory for JFS cache and such.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: mobybrick on 2010.05.10, 22:06:59
Yes, it should. But why static linking is not used, I do not know. If this was an pure OS/2 library then it would and OS/2 provides a static facility. The size of the program executables would grow (and overall, memory consumption would increase) and how items such as libc_sharedmem would work would need effort.

KLIBC needs a bit of an update IMHO, to fix some of the long-outstanding bugs and to work on the memory and pthreads situation.

Regards,
Moby.

Quote from: RobertM on 2010.05.10, 20:48:06
Quote from: mobybrick on 2010.04.25, 01:50:09
2. Building apps with static linking under GCC - reducing reliance on a shared DLL in memory. Whether or not this is feasible I don't know - but it would appear to solve other problems IMHO.

Regards,
Moby

Moby, do you think simply number 2 would alleviate these issues? I've gotten some info from RXU and am seeing the following:

After a few hours of running:
- Free Shared Memory: 44MB
- Free Memory in Private Arena that the process can allocate: 291MB

After shutting down services:
- Free Shared Memory: 245MB
- Free Memory in Private Arena that the process can allocate: 291MB

After restarting services:
- Free Shared Memory: 223MB
- Free Memory in Private Arena that the process can allocate: 291MB


The key being it seems that the private arena has a big chunk of available memory - currently more so than the shared arena starts with. If each (PHP, MySQL and Apache) had LIBC063 functionality statically linked, would that mean that EACH would have that 291MB available to it?

I am also suspecting that on most systems, the amount will be more... as I am allocating a bit of memory for JFS cache and such.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys stateme
Post by: RobertM on 2010.05.18, 16:51:01
NOTE 05/18/2010 @ 13:28EDT:
I have modified the script a little... I have now added the force shutdown and shutdown parameter to the MySQLAdmin shutdown statement to ensure that control from MySQLAdmin returns on completion.

I have also preceded all the other cmdline calls with the "@" symbol so they aren't echo'd to the console window.



Hey all,

Sorry this took so long for a very minor (programming wise) update, but here is the revised script including use of RXU to address low memory issues.

NOTES NOT IN THE SCRIPT:
- You MUST have RXU installed (find it on Hobbes, grab most recent version)
- You MUST either:
  (a) have it loaded and available via another script (I load all my REXX dlls via one script on system start) OR
  (b) add the code to load it into this script (standard method to load the REXX DLL functions, see RXU's documentation)

- This script will automatically shut down and restart MySQL and Apache in x number of hours (see notes in script)

- This script will check/monitor free shared memory space every 5 seconds and upon reaching y threshold, kill and restart MySQL and Apache.

- Hours before restart and minimum free memory before untimed restart are defined in script

- Script now shows time before restart (in seconds) as ###Sec Elapsed/###Sec to Restart - once #Sec Elapsed exceeds #Sec Restart, it will trigger a restart

- Script now shows (first line) amount of shared memory available when script was started

- Script now shows percentage remaining of shared memory every 5 seconds. This percentage is based off what was available on script start (it is not based on max available of 512MB). Inotherwords, if this script started with 200MB available in the shared arena, and 100MB of that gets used, it will show 50% free.

- When I get a chance, I will make the display a bit neater, maybe even with a few bar graphs for mem usage and time, and will probably incorporate some routines to query MySQL and Apache for what their current load is. It may be a month before I get around to that though, as I am deep into helping plan the next Star Trek Phase 2 Shoot, and once I finish the planning work, will be away at the shoot until the third week in June.


/* Start and Control MySQL and Apache */
/*
    THIS SCRIPT IS IN MY C:\Scripts Directory
    You must modify this line to the location you place it on your system
*/
    ThisDir="C:\Scripts\"



/*
    Currently, this script does NO checking to see if it actually succeeds in trying to kill
    either Apache or MySQL. If it fails, it will still try to spawn new copies.
*/
    '@mode 80,8'

    rtc=SysCurState("OFF")
    dosrc = RxQuerySysInfo( 'RxSysInfoOrig.' )

    /* Define Daemon Locations */
        ApacheDir='D:\SERVERDAEMONS\APACHE2\'           /* DONT FORGET TRAILING SLASH */
        MySQLDir="D:\SERVERDAEMONS\MYSQL5\BIN\"         /* DONT FORGET TRAILING SLASH */
        ApacheLogsDir="D:\serverdaemons\apache2\logs\"  /* DONT FORGET TRAILING SLASH */
            /* This entry is where httpd.pid will be located */

    /* Define wait time before killing and restarting MySQL and Apache */
        xHours=3

    /* Define Minimum Avail Shared Memory Space */
        memMinimum=60000000

    /* SLEEP for 3 seconds to allow killing of the script if need be */
    /*
        rtc=SysSleep(3)
    */



    /* Detach and Run MySQL */
    /*
        REMOVE any parameters in the "start... MySQLD.EXE" line below that you do not use:
        I am simply too lazy to change my my.cnf file while I am testing what works on my system
    */
        '@start "MySQL Daemon" /c /b /min 'MySQLDir'MYSQLD.EXE --max_seeks_for_key=50 --max_connections=500 --bulk_insert_buffer_size=32M --key_buffer_size=48M --table_cache=128 --sort_buffer_size=16M --read_buffer_size=8M'

        rtc=SysCurPos(5,0)
        Say Left("-STARTED: MySQL Daemon",79)


    /* Detach and Run Apache */

        '@start "Apache" /c /b /min 'ApacheDir'STARTUP.CMD'
        rtc=SysCurPos(6,0)
        Say Left("-STARTED: Apache Daemon",79)


    /* Wait xHours until kill and restart of services */
        rtc=SysCurPos(0,0)
        Say "Waiting "xHours" hours from: "Time()"   Started with "RxSysInfoOrig.21" bytes free memory"
        SleepTime=60*60*xHours

        cSleep=0
        Do Forever
            rtc=SysSleep(5)
            cSleep=cSleep+5
            dosrc = RxQuerySysInfo( 'RxSysInfo.' )
            if RxSysInfo.M<memMinimum Then
            Do
                rtc=SysCurPos(2,0)
                Say Left("-WARNING: Memory Threshold Met ("RxSysInfo.21<memMinimum")",79," ")
                rtc=Beep(349,10)
                Leave
            End

            rtc=SysCurPos(1,0)
            Say Left("-Reset Timer: "Right(cSleep||"/"||SleepTime,Length(SleepTime)*2+4)" seconds. ",79," ")
            rtc=SysCurPos(2,0)
            Say Left("-Free memory: "RxSysInfo.21" bytes    ("Format((RxSysInfo.21/RxSysInfoOrig.21)*100,3,1)"% free)",79," ")



            If cSleep=35 Then
            Do
                rtc=SysCurPos(6,0)
                Say Left(" ",79)

            End

            If cSleep=30 Then
            Do
                rtc=SysCurPos(5,0)
                Say Left(" ",79)

            End

            If cSleep>SleepTime Then
            Do
                Leave
            End


        End


    /* Request proper shutdown of MySQL and Apache */

        rtc=Kill_Apache()
        rtc=Kill_MySQL()

        /*
            BOTH of these are called WITHIN this REXX procedure and MUST be called from it
            - If you try to detach these, then if there is a problem (out of resources) the
              detached processes will NOT be started and the resources will not be freed.

              Inotherwords, nothing will happen at all except this will exit with two out of
              resources errors and the server daemons will not be restarted.

            - I have moved the Apache "Shutdown" script into this as a "function"
              You need to ensure that KILL is available in your path statement
              - Inotherwords, you need to add an entry to the config.sys path statement
                or ensure you have placed KILL.EXE in an already defined locations such as
                \OS2\BIN
        */


    /*
        Assumes both services have been killed.
        At this point, to ensure all resources are freed (including those used by this script)
        we will spawn another copy of this script which will wait 3 seconds (as defined above)
        before restarting the services in their own process spaces.

        This script will then exit 3 seconds later to free any resources it is using.

        While this script should not be using much in the line of resources, I have found that
        on ObjectREXX (especially the version shipped with eCS 1.2MR) every REXX script seems to
        leak resources the longer it runs.

        Thus, killing and restarting this script, instead of just looping forever, will alleviate
        those ObjectREXX issues.
    */




    /* Time to restart this script in a new session and then exit and close this session */
        rtc=Restart_AM()
        /*
            You may wish to remove "/min" from the above to allow you to easily find and kill this
            script in the 3 second window it gives you in the event it gets stuck in a restart loop.

            While that shouldnt happen, it's just a thought.

            To properly kill things if a loop occurs or there is another problem, find and kill THIS
            script first, then properly shut down MySQL and kill Apache - then restart this script.
        */


        /*
        rtc=SysSleep(3)
        */

Exit
            /* This script exits here */





Kill_Apache:
    pid = linein(ApacheLogsDir"httpd.pid")
    '@kill.exe -TERM 'pid
Return 0


Kill_MySQL:
    '@mysqladmin shutdown --force --shutdown_timeout=30'
Return 0


Restart_AM:
    '@start /c /b 'ThisDir'Run_ApacheMysql.cmd'
Return 0


Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys stateme
Post by: RobertM on 2010.05.18, 20:35:41
Still needed:
-Error checking (especially to ensure initial values are not too low... (such as in my case if I were to set the memory restart trigger at 300MB, when the system only has 280MB free after boot and loading of the support DLLs and firewall)).

-Fancier interface

-More information on the interface

-Perhaps more detailed information on both daemons and/or on the TCP/IP stack itself
(that may come in handy if you have a server that gets attacked or flooded from time to time - we had such an attempt yesterday where one IP continued to open (and hold open) as many http requests as possible - for some reason, Apache didnt do anything about it, though I am sure I have a per client connection limit in place - or am misreading the setting in the config file).

-"Theoretically" (ie: when I get around to it), the script can automagically add such IP entries to the firewall, force the firewall to reload it's rules and create a log for review.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.07.09, 06:31:08
Hi all,

I HOPE to have some very good news regarding this problem, very soon.

First, don't thank me: I will just be the messenger - all the work involved is being done by someone else. I'll let you know how things go once we do some testing. If all works out, (assuming the person doesnt post something about this before then), I'll let you all know who we have to thank for it.

Best,
Robert
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.07.10, 18:32:54
Quote from: RobertM on 2010.07.09, 06:31:08
Hi all,

I HOPE to have some very good news regarding this problem, very soon.
....


Could this be the elusive static-linking to fix lots of memory issues problem being finally solved?  Or an update to your shutdown/restart script?

If its the static linking, that would be fantastic!  Not to say the shutdown / restart script doesn't help, but it is still a work-around for the real issue.

Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.07.10, 19:57:30
Quote from: Fahrvenugen on 2010.07.10, 18:32:54
Quote from: RobertM on 2010.07.09, 06:31:08
Hi all,

I HOPE to have some very good news regarding this problem, very soon.
....


Could this be the elusive static-linking to fix lots of memory issues problem being finally solved?  Or an update to your shutdown/restart script?

If its the static linking, that would be fantastic!  Not to say the shutdown / restart script doesn't help, but it is still a work-around for the real issue.



LIBC063 apparently already supports using high memory - IF the apps that call it are compiled to request it for their shared memory space.  :)

MySQL and PHP (most recent version, at the least) are already compiled to request high memory...  :)
- Make sure you've upgraded your PHP and modPHP stuff in the Apache modules directory

Now, Apache on the other hand...
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.07.11, 01:07:20
Quote from: Fahrvenugen on 2010.07.10, 18:32:54

If its the static linking, that would be fantastic!  Not to say the shutdown / restart script doesn't help, but it is still a work-around for the real issue.



Things are looking VERY VERY good so far... and very snappy too! I think by Monday, I will have something very decent to post.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.07.11, 01:35:15
I'm anxiously awaiting Monday!

That does sound promising though.  I might be able to move my most troublesome website running - currently running on an XP box with Apache / PHP / MySQL (and that's the only thing running on that box, and it only has 1 website on it, so essentially its a dedicated machine type of thing currently) back to an OS/2 platform!

And why is it troublesome?  Because its running on the multi-user version of Wordpress (even though we havn't really used the multi-user thing yet) and there's something in one of its PHP scripts that every so often grabs huge amounts of shared memory and was causing my OS/2 server to crash frequently. I had to get it off that server, and decided to try it on an XP  machine at the time.

Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.07.11, 05:04:42
Hey gang,

So far, everything looks wonderful! Our benefactor in this endeavor is (for those who havent guessed) Paul Smedley! (Thanks Paul!!!)

For those of you wanting to test this, here's the link to the new build and everything else you need:



If you'd like to test it:
If you DONT have Apache v2.2.15 already installed, grab this package:

   *  http://smedley.info/httpd-2.2.15-os2-20100711.zip

IF you are upgrading from an older version of Apache, then make sure you back up your config files first, as xcopying the files over an existing installation will overwrite them.

If you DO have v2.2.15 installed, you can either replace it with the package above, or download this from Paul's site and copy the files in the zip file into your Apache\bin directory.

   * http://download.smedley.info/httpd-testfix-20100710.zip

   - Changes are in the readme

Be sure you have updated MySQL and PHP with these versions:

   * MySQL:
     http://os2ports.smedley.info/index.php?page=mysql-5-1

     (besides LIBC063, it also requires GCC444, located here:
         o http://os2ports.smedley.info/index.php?page=gcc44

   * PHP:
     http://os2ports.smedley.info/index.php?page=php-5

Copy the modphp.dll file in the PHP5\apache directory into Apache's modules directory, and ensure there are no other copies of older versions of modphp.dll or PHP/Apache/MySQL anywhere referenced in your path statements.

That should be it!

The only odd thing I have run in to is that I cannot start MySQL and Apache at the "same" time - I'm still running my little memory monitoring script (though it's set for 24 hours now), and all I had to do was add a one second pause between starting MySQL and Apache (possibly because Apache is looking for/loading some SQL service/dll that isnt yet initialized or is locked during MySQL's startup? Not sure why).

Otherwise, all is working flawlessly for me. After daemon initialization, no usage of the LMA at all (even after... 6+ hours of relatively heavy use serving a big SMF forum), no hangs, no delays... and possibly faster performance (the performance is definitely faster, but I am not sure if that's because of the high memory changes to Apache, or because v2.2.15 is simply faster than the older v2.2.8 I had been running).

Thanks again Paul!

Best,
Robert
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Paul Smedley on 2010.07.18, 11:11:18
Feedback from anyone else, other than Robert?
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: ivan on 2010.07.18, 13:15:02
Maybe if I knew what it was all about but page 3 of this thread is just a blank with only the masthead.

ivan
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Bill on 2010.07.18, 15:38:44
Actually,  pages 1, 2 and 3 are blank.  Only this page (4) has any entries

regards  Bill
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Ben on 2010.07.18, 16:17:04
You have to bypass the CMS and go directly to the forum to view a thread under these circumstances.

I'd post the link here, but I believe that to be against the wishes of the maintainers.

However, it is not difficult to figure it out and the first choice most likely, will be the correct one.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: ivan on 2010.07.18, 17:43:48
Hi Ben,

I do go direct to the forum and as Bill says the first three pages are blank.

Maybe the maintainers will eventually fix the problem.

ivan
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.07.18, 18:54:07
Quote from: Ben on 2010.07.18, 16:17:04
You have to bypass the CMS and go directly to the forum to view a thread under these circumstances.

I'd post the link here, but I believe that to be against the wishes of the maintainers.

However, it is not difficult to figure it out and the first choice most likely, will be the correct one.

It definitely is not. The direct link is this:
http://www.os2world.com/forum/index.php/topic,2297.0.html

We are aware of the problem and have tried numerous things to fix it (to no avail).

Best,
Robert
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.07.18, 22:26:08
Paul,

So far I havn't had any issues with the new Apache in the testing I've done on my home machine, it seems to be working well.  However...

I havn't yet put the website that I was having major issues with on to this build of Apache.  This is because:

1.  That website does have a high level of traffic, I need to schedule any time I'm going to take it offline and be ready to go with getting it back online to minimize downtime

2.  In order to accomplish this, I'm going to need to build an OS/2 server running this version of Apache, PHP, MySQL, and install the website on it.  Ensure it is working, and then switch from the current machine over to it, and see what happens.

I do have 2 OS/2 servers at that location, however they're both mission-critical, so I'm not about to test this on either of  those machines.  If I'm able to confirm this solves the issue on the stand-alone machine running just that website then I'll look at putting it on to one of those 2 servers, but for now, I'm going to play it safe.

I'll definitely let you know when I have a chance to do more testing, but so far things are looking good!

Thanks!
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.07.18, 23:12:38
130,000 PHP generated page views, 200 new posts... stable so far.

Rob
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Fahrvenugen on 2010.07.19, 17:02:26
Quote from: RobertM on 2010.07.18, 23:12:38
130,000 PHP generated page views, 200 new posts... stable so far.

Rob

Rob,

Are you still running your script to shut down / restart MySQL / Apache on a regular basis, or have you turned that off?

Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2010.07.19, 17:42:02
Quote from: Fahrvenugen on 2010.07.19, 17:02:26
Quote from: RobertM on 2010.07.18, 23:12:38
130,000 PHP generated page views, 200 new posts... stable so far.

Rob

Rob,

Are you still running your script to shut down / restart MySQL / Apache on a regular basis, or have you turned that off?



It is currently set for every 24 hours - which I am about to extend to 48, then 96 and so on. I'd never get much farther than 4 hours before.

Currently, the server is 21 hours in (ie: 3 hours before daemon restart) and is still at 91% free low memory (started at 91.3%, now at 91.4%). So, in 21 hours of roughly 14,000 pageviews (16,000 per day) at an average 15 MySQL/PHP requests each (SMF uses 12-20 per page) - or 210,000 MySQL/PHP requests in the last 21 hours... all seems to be running well on the Apache end.

I have been having MySQL problems, but that's unrelated to this Apache build (and may be an error on my part for not completing the upgrade of various eCS components). MySQL just goes away from time to time. I've revised my script to monitor MySQL and restart it if it goes away, which has resolved that.

I seriously think that's an SMF/MySQL issue and not related to OS/2 or wAMP, because the same exact thing happened on the Linux box the forums were originally running on.

The one thing I do know is this... AMP on OS/2 definitely is outperforming AMP on Linux. We arent running a 4 way turned 8 way SMP box for these forums (as they were when they were on the Linux box). We are running a single core P4 2.4 GHz CPU with only 2GB RAM.

Anyway, I will post the new script with the MySQL monitoring section added sometime later...

Best,
Robert
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Ben on 2010.07.19, 21:27:08
Quote from: RobertM on 2010.07.18, 18:54:07

We are aware of the problem and have tried numerous things to fix it (to no avail).

I had that problem too and I believe I pointed it out when I recommended the Joomla/SMF combination, way back when, to OS2world.

The problem I resolved by getting rid of Joomla and replacing it with Elxis... a variant. Don't ask me why it went away, but it did... and some other problems too.

I got rid of the bridge later on as well and switched to the wrapper.

There is an Elxis integration tool that works with SMF and new posts --not the same as a bridge--, (though I haven't tested it, no one has reported any errors... but it has been a while since I checked).

Food for thought perhaps... though the combination may not suit the needs of this site, it has been very reliable for me now for a few years.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Ben on 2010.07.25, 12:55:31
And I'll just add this...

Back when I was using Joomla, (and Mambo), getting blank pages was not limited to SMF through the bridge.

Some days some 'main site' pages would be served blank and would persist for a time, another day they would be served just fine. Sometimes tweaking the article and saving it would fix it, other times it wouldn't.

While this was far more prevalent with SMF through the bridge, it was certainly not limited to it.

SMF by itself never had this problem. The obvious commonality here being the CMS.

Just a crumb more of data.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: abwillis on 2011.06.13, 22:13:55
Quote from: RobertM on 2010.07.18, 18:54:07
Quote from: Ben on 2010.07.18, 16:17:04
You have to bypass the CMS and go directly to the forum to view a thread under these circumstances.

I'd post the link here, but I believe that to be against the wishes of the maintainers.

However, it is not difficult to figure it out and the first choice most likely, will be the correct one.

It definitely is not. The direct link is this:
http://www.os2world.com/forum/index.php/topic,2297.0.html

We are aware of the problem and have tried numerous things to fix it (to no avail).

Best,
Robert

I am trying to view page three of this thread but even going directly through the forum page it is blank for me...  I am looking in particular for the script if it was posted for restarting the Mysql/Apache.
Andy
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2011.10.11, 04:32:33
Andy, did you ever find the script? Do you still need it? If so, I can re-post it. I think I'd added a couple tweaks to it since last posting anyway.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: abwillis on 2011.10.11, 06:10:24
I still can't view it... I was pointing someone else to it and I am not sure if he was able to view it or not so I would appreciate the a re-post.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2011.10.11, 06:22:29
Will do... running around today, but will get it posted tomorrow. On a related note, for some odd reason (perhaps something I installed?) I found the following in my config.sys file:

 SET SHELLHANDLESINC=20

THAT bit of nastiness was causing all sorts of issues when I tried adding a new virtual host to the virtual hosts config file. A (paraphrased, not in front of the machine) "cannot load mime.types - file not found" issue. Remove the new host, works. Add any new host, Apache fails with that error. I'm wondering (ie: strongly suspecting) that it was also the cause of the occasional blank page being served. And possibly some of the weird MySQL corruption issues of the past (which increasing thread concurrency minimized but did not eliminate). I've changed the value to 80, added back the new domain name (and others) and things are running smoothly.

So, note to those running Apache/2:
Check your config.sys file for that statement. The following is true:
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: abwillis on 2011.10.11, 07:10:01
 SET SHELLHANDLESINC=80 is what I have used as a minimum setting even on my laptops for many years but I can't say for certain on the systems we were looking at this thread about.  Just double checked and was told he's been running 120 for some time now.
Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: Ben on 2011.10.11, 13:55:35

I've had it set at SET SHELLHANDLESINC=20 for some time now, without any issues.

I'm running about five sites from my server.

I changed it to 80 as you suggested and I'll see if anything changes.

If you don't hear back from me then nothing noticeable occurred.

Title: Re: (Apache/PHP/MySQL related) Looking for info on PROCESSES= config.sys statement
Post by: RobertM on 2011.10.11, 21:11:24
I think it becomes a problem only when:
(1) you pass 13-16 sites and/or
(2) PHP or other spawned handlers hit the 13-16 handle limit of what remains of the 20