OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Applications => Topic started by: Dave Yeo on December 16, 2018, 06:16:19 pm

Title: mozturbo
Post by: Dave Yeo on December 16, 2018, 06:16:19 pm
I've updated the old mozturbo using the code from SM-1.19 for Firefox, SeaMonkey and Thunderbird, 3 binaries, one package. https://bitbucket.org/dryeo/mozturbo/downloads/ (https://bitbucket.org/dryeo/mozturbo/downloads/)
These will preload the DLLs used by the various packages. The main reason for this is that our kernel has a bug, see https://mantis.arcanoae.com/view.php?id=2194 (https://mantis.arcanoae.com/view.php?id=2194). Basically when using high memory and closing an app, the memory that the DLLs allocated is not unallocated, leaving zombie holes in memory and eventually running out of high memory. Even without using high memory, this may help memory fragmentation.
Install with the browser or email client closed. Simplest is to put it in the working directory where the DLLs are located though it has an option to pass the location to it, create a program object, put in your startup folder with the -l (L) parameter and start it or better, reboot. Use programs as usual. You can use Theseus to keep track.

I've had good results using it with SM and TB, haven't tested too much with FF. Others have reported some problems with FF, low memory usage or not fixing slowness on restart.
Anyways shouldn't hurt, it has a command to unload the DLLs if needed. Run without parameters to see options.

Should work with most versions of Mozilla, targeted at 45ESR and 38ESR
Title: Re: mozturbo
Post by: Pete on December 16, 2018, 09:39:09 pm
Hi Dave

... The main reason for this is that our kernel has a bug, see https://mantis.arcanoae.com/view.php?id=2194 (https://mantis.arcanoae.com/view.php?id=2194).


I can only get an Access Denied page when trying the above url - even when logged in. Is anyone else having problems accessing this url?


Regards

Pete
Title: Re: mozturbo
Post by: Dave Yeo on December 16, 2018, 10:00:34 pm
Quite possible you don't have permission. It's just Rich's description of the problem with some logs and screenshots from Theseus showing the problem.
You can load Theseus, highlight a process and choose under the process menu, Shared Object Summary and after opening and closing and reopening an app with DLLs marked to load high, see the results, basically chunks of allocated memory with no owner but the same size as the DLL currently loaded. For example, though the zombie memory was from SM 2.35, first area after the first Free area was probably xul.dll
Code: [Select]
  Object Allocated Committed   Present   Swapped
 address    memory    memory    memory    memory  Description
8BF31000  0000F000                                 Free
8BF40000  03E1F000  00000000  00000000  00000000 
8FD5F000  00001000                                 Free
8FD60000  000A0000  00000000  00000000  00000000 
8FE00000  00704000  00000000  00000000  00000000 
90504000  0000C000                                 Free
90510000  00080000  00000000  00000000  00000000 
90590000  00018000  00000000  00000000  00000000 
905A8000  00008000                                 Free
905B0000  000D1000  00000000  00000000  00000000 
90681000  0000F000                                 Free
90690000  00002000  00000000  00000000  00000000 
90692000  0000E000                                 Free
906A0000  00005000  00000000  00000000  00000000 
906A5000  0000B000                                 Free
906B0000  00001000  00000000  00000000  00000000 
906B1000  0000F000                                 Free
906C0000  00037000  00000000  00000000  00000000 
906F7000  00009000                                 Free
90700000  00001000  00000000  00000000  00000000 
  Object Allocated Committed   Present   Swapped
 address    memory    memory    memory    memory  Description
90701000  0000F000                                 Free
90710000  03E1F000  00000000  00000000  00000000 
9452F000  00001000                                 Free
94530000  000A0000  00000000  00000000  00000000 
945D0000  006BC000  006BC000  006BC000  00000000  SUITE    #0001 (shared code)
94C8C000  00004000                                 Free
94C90000  0006A000  00000000  00000000  00000000  LGPLLIBS #0001 (shared code)
94CFA000  00006000                                 Free
94D00000  000A7000  00000000  00000000  00000000  MOZSQLT3 #0001 (shared code)
94DA7000  00009000                                 Free
94DB0000  039A0000  00000000  00000000  00000000  XUL      #0001 (shared code)
Title: Re: mozturbo
Post by: OS4User on December 16, 2018, 10:10:58 pm
Basically when using high memory and closing an app, the memory that the DLLs allocated is not unallocated

VMM releases only physic mapping  but address space remains in use. (This bug is fixed in OS/4 long back).
Title: Re: mozturbo
Post by: Dave Yeo on December 17, 2018, 06:12:54 am
Note there is a bug in ffturbo, missing DLL. I'll upload a new build later. The others should be fine.
Title: Re: mozturbo
Post by: Dave Yeo on December 17, 2018, 04:20:57 pm
New version uploaded, only change is for Firefox users, namely adding a missing DLL in ffturbo.
Title: Re: mozturbo
Post by: xynixme on December 19, 2018, 10:48:56 am
Simplest is to put it in the working directory where the DLLs are located though it has an option to pass the location to it

I presume you mean the non-LIBPATH home directory of XUL.DLL.

Is it recommended to use high memory in general, because all qualifying FF/TB or SM DLLs will always be loaded, even when you're not going to use FF/SM/TB while working with some large database?

I'll try to find the minimum memory requirements of just using *TURBO.EXE, without actually using the related full product. I can restrict installs to >= 1 GiB of RAM, but if it just loads DLLs then 256 MiB may be enough too. To simplify installs, one tends to not actually use FF/SM/TB that frequently with 256 MiB of RAM.

Later you could consider to ship *TURBO.EXEs with SM/TB updates, if any, since the Mozilla browser installed by the OS has an own MOZTURBO.EXE in its own main directory too. Mine's x:\PROGRAMS\MOZILLA\MOZTURBO.EXE. Just to promote consistency, and using it will remain optional.
Title: Re: mozturbo
Post by: xynixme on December 21, 2018, 05:10:42 pm
Is it recommended to use high memory in general, because all qualifying FF/TB or SM DLLs will always be loaded, even when you're not going to use FF/SM/TB while working with some large database?

Can it be compiled as a PM app, without the FCF_TASKLIST flag? You may be able to use Van Wijk's PMTPOSD2 (stealth mode) as sample code.
Title: Re: mozturbo
Post by: Dave Yeo on December 23, 2018, 05:06:22 am
Hi André,
Sorry for not responding sooner, no power for a few days due to big wind.
Yes, you should never have any of the Mozilla apps on LIBPATH, so home directory of xul.dll.
Loading the DLLs into high memory is a users choice. Generally a good idea due to the shortage of address space in low memory. Note that you need a recent kernel ideally due to bugs in older kernels. Some or most of these bugs may happen when unloading the DLLs, so mozturbo maybe a workaround for that as well.
If you do load into high memory, when unloading, the kernel will not unallocate the memory, so you lose memory anyways, even if working on large databases etc.
It does jsut load the DLLs and I have an update to push that uses less memory but doesn't work as well as a turbo launcher, there's a switch to bring back the current way it works. I'll probably upload tomorrow after considering whether to make it a PM app. I'll look at Van Wijk's PMTPOSD2 later.
BTW, with mozturbo running and doing various tests which involved unloading and reloading, I managed 8 days of uptime, usually the system traps or locks up after 2-3 days. Been years since I've had much more then 3 days uptime.
Title: Re: mozturbo
Post by: Rich Walsh on December 23, 2018, 04:36:35 pm
[...] after considering whether to make it a PM app.

Please don't! Currently it can be launched as a background daemon (e.g. "detach ffturbo") which uses almost no memory on its own. I can't see how making it a PM app adds anything useful.

Creating a couple of program objects to launch and terminate it makes more sense to me:
Path and file name: 'cmd.exe'
Parameters: '/c detach ffturbo -l'  [..or..]   '/c ffturbo -u'
Working directory: [your FF directory]
Title: Re: mozturbo
Post by: Dave Yeo on December 25, 2018, 05:07:08 am
Updated to v0.5. Default is now not to force load the modules, saving quite a bit of memory at the cost of turbo functionality. Thanks to Rich for pointing this out. Added the -t parameter for the old behaviour.
I don't anticipate any more binary changes to the programs.
https://bitbucket.org/dryeo/mozturbo/downloads/mozturbo-v0.5.zip (https://bitbucket.org/dryeo/mozturbo/downloads/mozturbo-v0.5.zip)
Title: Re: mozturbo
Post by: xynixme on December 27, 2018, 12:49:26 pm
Please don't! Currently it can be launched as a background daemon (e.g. "detach ffturbo") which uses almost no memory on its own. I can't see how making it a PM app adds anything useful.

Creating a couple of program objects to launch and terminate it makes more sense to me:
No protests obviously, as long as results are the same: an user doesn't see it in any default tasklist and can shutdown the system without an extra dialog. Without PM all *TURBO.EXEs will work the same way too, AFAICT. Including the original one.
Title: Re: mozturbo
Post by: xynixme on December 27, 2018, 01:04:33 pm
I'll stick to the default settings w.r.t. high or low memory.

BTW, with mozturbo running and doing various tests which involved unloading and reloading, I managed 8 days of uptime, usually the system traps or locks up after 2-3 days. Been years since I've had much more then 3 days uptime.

Probably no surprise, but according to Theseus it seems okay to use it with systems with <= 512 MiB of RAM too. One tends to not use e.g. SM that often then, but one also tends to not use other hungry apps. Which allows me to use the *TURBOs everywhere.

PMTPOSD2 would be hardly more than a skeleton app with too many bones, but it has a matching stealth mode (no tasklist, no extra shutdown dialog). I'll try DETACH instead, if that fully hides it as well.

Thanks for the smart solutions!
Title: Re: mozturbo
Post by: Dariusz Piatkowski on December 27, 2018, 04:40:12 pm
Hi Dave!

Updated to v0.5. Default is now not to force load the modules, saving quite a bit of memory at the cost of turbo functionality. Thanks to Rich for pointing this out...

Alright...so what is the benefit / functionality being delivered with this new default setting? I mean after all, isn't the whole point to pre-load the DLLs and prevent their un-loading during program shut-down?

Sorry, I simply missed it and/or do not understand what the utility would do otherwise if it is not actualy doing the DLL load.

Also, as a follow-up request for clarification, what happens to other DLLs that do not reside in the main directory? For FF I am specifically thinking:

Code: [Select]
The volume label in drive G is OS2.
The Volume Serial Number is 45B6:75C3.

Directory of G:\apps\tcpip\firefox\browser\components

12-23-18   6:36p    369383         124  brwsrcmp.dll
        1 file(s)     369383 bytes used

Directory of G:\apps\tcpip\firefox\gmp-clearkey\0.1

12-23-18   6:20p    306136         124  clearkey.dll
        1 file(s)     306136 bytes used


Are these left-out?

Thanks!
Title: Re: mozturbo
Post by: Dave Yeo on December 27, 2018, 06:34:03 pm
Hi André, using the FCF_TASKLIST flag means a full PM app, I believe it applies to the frame, something that would be overkill and I'm not interested in. The source is there if you decide to do it yourself, it would be fairly easy as far as I can see.
Being detached, the process won't show up in the window list but may still need to be shutdown on exit, I haven't tested. Most of us are running some version of Xworkplace as it is included in ArcaOS and at least later versions of eCS as well as running fine on older versions of Warp, or at least Warp v4. It can ignore processes on shutdown so a one time setting and forget about it. At the worst, you can create a program object to unload the daemon and click it before shutting down.
Title: Re: mozturbo
Post by: Dave Yeo on December 27, 2018, 06:47:22 pm
Hi Dariusz. The difference is in how much of the DLLs get preloaded. With the new setting, the DLLs just have a smaller memory footprint. You can test it yourself using Theseus. Choose a process, any process seems to work as we're looking at shared memory, and then choose Process-->Shared Object Summary and scroll down towards the bottom of the new window where high memory is and observe how much memory xul.dll for example is using.
As for brwscmp.dll, I missed that with the first release, which I just deleted. Don't use that release with Firefox as the missing brwscmp.dll caused buggy behaviour with Firefoxes memory. SM and TB were fine with that release.
Clearkey.dll shouldn't even be installed as it does nothing on our platform currently and probably never will. It's for DRM managing H264 files, things like Netflix I guess. See https://wiki.mozilla.org/GeckoMediaPlugins (https://wiki.mozilla.org/GeckoMediaPlugins). The fact it is never loaded can be verified with Theseus or simply by moving it out of the way with Firefox running. Even if it got dynamically loaded like avcodec (FFmpeg), unless it is marked to load high, it wouldn't particularly matter.
Title: Re: mozturbo
Post by: Dariusz Piatkowski on December 28, 2018, 02:15:37 am
...As for brwscmp.dll, I missed that with the first release, which I just deleted. Don't use that release with Firefox as the missing brwscmp.dll caused buggy behaviour with Firefoxes memory. SM and TB were fine with that release...

Ahhh...indeed, I remember one of your emails on our tester list about this, sorry to bring that one back, I am using your latest release now and was previously using the fixed one which picked that DLL as well.

Alright, I will take a look at the default and the '-t' optional switch and will post the results I'm seeing on my machine.
Title: Re: mozturbo
Post by: Doug Bissett on January 04, 2019, 06:11:13 am
Quote
Updated to v0.5. Default is now not to force load the modules, saving quite a bit of memory at the cost of turbo functionality. Thanks to Rich for pointing this out. Added the -t parameter for the old behaviour.
I don't anticipate any more binary changes to the programs.

I just want to report, that the Firefox version is working well (no parameters). I had to reboot, for a different problem, today, after 7 days up time (first time, in a long time). Something is still eating lower shared memory, but it seems to taper off, after a few days.

Thank you.
Title: Re: mozturbo
Post by: Dave Yeo on January 05, 2019, 01:02:18 am
Good to hear. I was on the 8th day of uptime before the wind storm took out the power. That was with restarting mozturbo a few times for testing purposes. It seems like it has been decades since I got that much uptime.
Be interesting to know where you're lower memory is going. Looking here, the DLLs still use some lower memory, XUL and such grab some private memory in the shared arena for some reason and of course all the supporting DLLs load in low memory and seem to use some for private purposes as well.
Then there are things like FC/2, which when I started it up, used quite a bit of low memory, 30 odd MBs. Looking, it all seemed to be claimed by PMMERGE.

I also marked some of the supporting DLLs to load high, namely icudt.dll, icuin.dll, icuuc.dll, hunspell10.dll, libvpx4.dll as they used the most and seemed safe. The NSPR and NSS DLLs are probably safe as well. Others, I'm reluctant to mark high as I don't think all programs can use them if marked high. I marked Cairo at one point and got weird results from the screensaver and even mozturbo gave me weird results until I added --Zhigh-mem to the LDFLAGS so have to be careful what is marked high.
They all need unlocking before marking as well as mozturbo loads them all and a restart of mozturbo or reboot is required to actually use them.
Have to remember that if any of these DLLs are upgraded, they'll also be locked.
Title: Re: mozturbo
Post by: Doug Bissett on January 05, 2019, 07:24:05 am
Quote
Be interesting to know where you're lower memory is going. Looking here, the DLLs still use some lower memory, XUL and such grab some private memory in the shared arena for some reason and of course all the supporting DLLs load in low memory and seem to use some for private purposes as well.
Then there are things like FC/2, which when I started it up, used quite a bit of low memory, 30 odd MBs. Looking, it all seemed to be claimed by PMMERGE.

I haven't taken the time to try to figure out where the lower shared memory is going. Upper shared memory is also decreasing, but that isn't as serious, because it ends up around 1 GB left, while lower shared memory goes as low as 48 meg (when it continues to run). It seems that 100 meg of shared memory can disappear very quickly, sometimes. The main user, that I know about, is VBOX 5.0.6, which uses about 100 meg of lower shared memory. Once it gets itself running, it doesn't use any more, and it seems to return all of it when it terminates. The problem is, that other programs may need lower shared memory, while VBOX is running, and that can cause problems. I did try adjusting the VBOX DLLs to load high, but that didn't work, and I haven't tried again (yet).

Quote
I also marked some of the supporting DLLs to load high, namely icudt.dll, icuin.dll, icuuc.dll, hunspell10.dll, libvpx4.dll as they used the most and seemed safe. The NSPR and NSS DLLs are probably safe as well. Others, I'm reluctant to mark high as I don't think all programs can use them if marked high. I marked Cairo at one point and got weird results from the screensaver and even mozturbo gave me weird results until I added --Zhigh-mem to the LDFLAGS so have to be careful what is marked high.

Common DLLs, used by many, different, programs are probably not going to work well, if marked high. Anything used by only a single program (or perhaps a suite of related programs) is probably okay, as long as all of the programs can handle it. The main problem is, that there could be one thing, that isn't used much, that will fail. By the time that happens, the user will have forgotten what they did, and the relationship may not be obvious.

Oh well, it is still a step in the right direction, until somebody can figure out how to fix the problem properly.

Quote
They all need unlocking before marking as well as mozturbo loads them all and a restart of mozturbo or reboot is required to actually use them.
Have to remember that if any of these DLLs are upgraded, they'll also be locked.

Unloading MozTurbo would seem to be asking for trouble, since the main purpose is to keep the DLLs loaded, to prevent the DLL unload problem. I expect that any future updates would be done with RPM/YUM, and that seems to handle locked files okay. A reboot is probably the best way to handle that, but I am not sure how the user could be told to do a reboot (which is probably a good idea after updating those things anyway).
Title: Re: mozturbo
Post by: xynixme on January 05, 2019, 10:28:51 am
SMTURBO.EXE -T wasn't running anymore after several failing attempts to start SEAMONKEY.EXE, each resulting in a SYS2070. I was able to start SEAMONKEY.EXE after killing MOZTURBO.EXE -L as well, but that may be unrelated. SEAMONKEY.EXE was the first started, related product after a cold boot (loading was finished, I was editing a file before opening the browser).

01-05-2019  10:16:05  SYS2070  PID 0039  TID 0001  Slot 009a
D:\SEAMONKEY\SEAMONKEY.EXE
XUL->PLDS4.26
182

FWIW.
Title: Re: mozturbo
Post by: xynixme on January 05, 2019, 11:17:42 am
The main user, that I know about, is VBOX 5.0.6, which uses about 100 meg of lower shared memory.
FWIW: Object Rexx' file-I/O, otherwise a better Rexx interpreter, is a good way to run out of shared memory too. Until a WPS reset. I've never checked if ORexx would beat ~100 MiB, though.
Title: Re: mozturbo
Post by: xynixme on January 05, 2019, 11:34:06 am
using the FCF_TASKLIST flag means a full PM app, I believe it applies to the frame, something that would be overkill
A hidden frame, FWIW. So basicly the same effects as Rich's DETACH, regardless of the abuse of democracy by claiming "most of us are using ...": not visible, no shutdown dialogs, no requirements of requirements to avoid useless dialogs. ;-)

Which slower hardware and DETACH it'll appear in whatever task list a few people are using for a few zilliseconds, but that's it. It may be possible to hide selected apps, but that's not required with solutions like (not) FCF_TASKLIST or DETACH. DETACH is better, since there is no meaningful PM UI.

I've never tried to replace/upgrade eCenter bij xCenter, so apparently I'm not using what "most of us are using" indeed. That rather popular assumption is right. :-)
Title: Re: mozturbo
Post by: xynixme on January 05, 2019, 11:59:06 am
I also marked some of the supporting DLLs to load high, namely icudt.dll, icuin.dll, icuuc.dll, hunspell10.dll, libvpx4.dll as they used the most and seemed safe.
Regarding HUNSPEL0.DLL (8.3 DLL file name): I've just installed this DLL and no other files. So mine doesn't load and expand any dictionary files or other files, if that's what HUNSPEL0.DLL would be doing with your RAM. As such FF/SM only requires HUNSPEL0.DLL of that package, which doesn't appear to be a large file.

Nevertheless I'll mark HUNSPEL0.DLL as high, since it seems to be safe and I'm pretty sure no other app uses it. It just has to be there, as some useless technical requirement.
Title: Re: mozturbo
Post by: Doug Bissett on January 05, 2019, 06:06:58 pm
Quote
FWIW: Object Rexx' file-I/O, otherwise a better Rexx interpreter, is a good way to run out of shared memory too. Until a WPS reset. I've never checked if ORexx would beat ~100 MiB, though.

OREXX is known to have some serious problems. It is recommended to avoid using it, when possible.
Title: Re: mozturbo
Post by: Dave Yeo on January 05, 2019, 06:28:24 pm
SMTURBO.EXE -T wasn't running anymore after several failing attempts to start SEAMONKEY.EXE, each resulting in a SYS2070. I was able to start SEAMONKEY.EXE after killing MOZTURBO.EXE -L as well, but that may be unrelated. SEAMONKEY.EXE was the first started, related product after a cold boot (loading was finished, I was editing a file before opening the browser).

01-05-2019  10:16:05  SYS2070  PID 0039  TID 0001  Slot 009a
D:\SEAMONKEY\SEAMONKEY.EXE
XUL->PLDS4.26
182

FWIW.

Interesting, it is a NSPR DLL. Do you have something running that also uses NSPR? Old version of a Mozilla App? Seems to me other stuff has used NSPR but can't think of any right now.
Mozturbo will load the other DLLs required by SeaMonkey etc and here it must have been loading the wrong one.
BTW, SeaMonkey doesn't use the Hunspell package but rather the one included with Mozilla so no need for it.
Title: Re: mozturbo
Post by: Dave Yeo on January 05, 2019, 07:22:12 pm
Hi Doug, unloading and reloading mozturbo a couple of times will just waste some address space. While better to avoid, it didn't hurt here besides having available upper address space shrink.
Seems that for vbox, it is the QT DLLs eating up lower address space and leaving some fragmentation when closed and perhaps a leak as on one quick test, pmmerge seems to be left allocating quite a bit of low memory.
This raises the question of whether the QT4 DLLs can be safely loaded high and then we'll also have the same question about the QT5 DLLs. In this 64bit world, memory usage of ported apps is likely to increase. One thing is that it would be easy to modify mozturbo into qtturbo.
Title: Re: mozturbo
Post by: Doug Bissett on January 05, 2019, 08:10:55 pm
Quote
Hi Doug, unloading and reloading mozturbo a couple of times will just waste some address space. While better to avoid, it didn't hurt here besides having available upper address space shrink.

I found that loading/unloading Firefox, while using the DLLs in high memory, worked, most of the time, but not always. Throw in the leakage, and it didn't take very many load/unload cycles to cause problems. MozTurbo seems to work around the worst of the bad things that Firefox was doing. Unfortunately, other things need the same treatment, and that could go on forever. Many things also need to be rebuilt, to accept loading DLLs high. All we can do, is find programs that are capable, and make sure that they load high, and don't unload DLLs (at least until the problem gets fixed, which may be never).

Quote
This raises the question of whether the QT4 DLLs can be safely loaded high and then we'll also have the same question about the QT5 DLLs.

QT is used by a number of programs. If ALL of them are capable, it should be okay. If not, it won't be okay. VBox seems to be well written, and it does return all of the used shared memory (according to Above512, which isn't really accurate). The big question, that I have, is why is it using so much shared memory? I am pretty sure that it isn't all DLLs, and at least some of it could probably be loaded high.

Lots of experimenting to do.  :)
Title: Re: mozturbo
Post by: David McKenna on January 05, 2019, 08:55:16 pm
Hi Dave -

  Thanks for the SMTurbo thing - it seems to have made my system much more stable in general, and like others have said, uptime for SeaMonkey is much longer.

  Just thinking out loud... would it be possible to alter SMTurbo into a more generic 'Turbo' program that uses a text file containing a list of DLL's that can be loaded at startup (instead of hard coded)? Might be helpful with other sub-system DLL's that get loaded/unloaded over time...

Regards,
Title: Re: mozturbo
Post by: Dave Yeo on January 05, 2019, 09:25:38 pm
  Just thinking out loud... would it be possible to alter SMTurbo into a more generic 'Turbo' program that uses a text file containing a list of DLL's that can be loaded at startup (instead of hard coded)? Might be helpful with other sub-system DLL's that get loaded/unloaded over time...

I don't see why not, especially if the DLLs are on LIBPATH (and perhaps BEGINLIBPATH if wrapped in a script).
I'll give it some thought.
Title: Re: mozturbo
Post by: xynixme on January 29, 2019, 09:38:09 am
I also marked some of the supporting DLLs to load high, namely icudt.dll, icuin.dll, icuuc.dll, hunspell10.dll, libvpx4.dll as they used the most and seemed safe. The NSPR and NSS DLLs are probably safe as well.
Is XUL.DLL the single other one (of  your "I also ...")?
Title: Re: mozturbo
Post by: Dave Yeo on January 29, 2019, 06:03:24 pm
Usually (sometimes out of laziness, just xul.dll) I've been marking all the DLLs included with the moz app to load code high. Note that "make package" leaves some DLLs marked readonly for some reason and they need to be have that attribute changed. "Make install" doesn't mark them readonly. My dists are made with "make package"
Title: Re: mozturbo
Post by: xynixme on January 31, 2019, 04:36:08 pm
icudt.dll, icuin.dll, icuuc.dll, hunspell10.dll, libvpx4.dll as they used the most and seemed safe.
FWIW, not important: with those and XUL.DLL marked as high, and MOZTURBO.EXE started with a.o. LIBPATHSTRICT=T, both SMTURBO.EXE -t (in WP_START) and SEAMONKEY.EXE produce a SYS2070 once. Even after a cold boot. Kill SMTURBO.EXE (and MOZTURBO.EXE), mark XUL.DLL as high (again), shutdown, power off, wait a few seconds, power on, wait until a.o. FFTURBO.EXE should have done its job, start SeaMonkey, ...

Code: [Select]
01-31-2019  16:13:05  SYS2070  PID 0034  TID 0001  Slot 0097
D:\SEAMONKEY\SMTURBO.EXE
XUL->PLDS4.26
182
------------------------------------------------------------

01-31-2019  16:13:35  SYS2070  PID 0036  TID 0001  Slot 0095
D:\SEAMONKEY\SEAMONKEY.EXE
XUL->PLDS4.26
182
Once, with a Pentium 4 machine. After a next warm reboot everything worked as expected. Always, so far. May be normal, it's the first time I'm really using this high-memory feature.

I'll mark the other suggested ones as high later (NSS/NSPR?). If the rare SYS2070s occur again, then I'll use a full XOR approach. SMTURBO XOR FFTURBO XOR MOZTURBO. Now it's (SMTURBO XOR FFTURBO) and MOZTURBO.


The eCS 1.2 UNI 104 kernel of a Pentium III IBM ThinkPad 600X (with SMTURBO.EXE -l) crashed, twice, when I tried to kill SMTURBO.EXE via CAD before marking XUL.DLLs of FF/SM as high (done, unlocking is an alternative without a crashing kernel). One of the kernel crashes may have been related to an USB flash drive.


So far, so good.
Title: Re: mozturbo
Post by: Dave Yeo on January 31, 2019, 06:45:25 pm
Weird that you're getting a sys2070 with LIBPATHSTRICT. Are you setting BEGINLIBPATH with uppercase Pathnames? If so, try lower casing them. FYI, the missing export is
_PL_SizeOfArenaPoolExcludingPool @26 RESIDENTNAME
which seems to have been introduced between 17 and 24.
Also, are you trying to mark SM1.19 DLLs to load high? This is unsupported and untested and may fail due to using an old version of os2safe.h (and who knows what else), but then again might work. Really it could be recompiled, but judging by how many casts I had to add to mozturbo, it is a bigger job then I feel like undertaking.
The kernel crashes are possibly kernel bugs in unloading DLLs marked to load high, a known issue with newer kernels being patched to fix some of these bugs. Another reason to keep them loaded. BTW, why use the uni kernel? The W4 kernel will give perhaps 10% better performance and ArcaOS doesn't even include the uni kernel as it doesn't add anything but overhead.

I've recompiled the old mozturbo with LIBPATHSTRICT if you want to test. This is only for use with SM 1.19 and is untested.

Edit, forgot to mention that I usually run all 3 turbo programs, ffturbo, smturbo and tbturbo without problems.
Title: Re: mozturbo
Post by: xynixme on February 05, 2019, 03:17:44 am
FWIW: the system load during boots seems to be too much for XWLAN. With a T42p I have to restart it to obtain a connection, which never was a problem before the *TURBO.EXEs were in use. Is a built-on delay of a few milliseconds be an option, instead of pumping DLLs to memory with all available power? I don't know if that could prevent the rare SYS2070s too. As if setting up a WLAN connection is time-critical, but it doesn't get enough time slices.

In the case of the ThinkPad T42p I'm using MOZTURBO.EXE -l (old, with LIBPATSTRICT setting) and FFTURBO.EXE -t. A slower T23 has the same problem.

Weird that you're getting a sys2070 with LIBPATHSTRICT. Are you setting BEGINLIBPATH with uppercase Pathnames? If so, try lower casing them. FYI, the missing export is _PL_SizeOfArenaPoolExcludingPool @26 RESIDENTNAME which seems to have been introduced between 17 and 24.
Yes, uppercased or a reality-matching mixed case. I've got a registered version of DragText, so I can edit the settings and change it to lower case.

The latest change was to mark XUL.DLLs as high too, after HUNSPEL0.DL (first test) and next the other generic DLLs. If that works fine everywhere (a.o. old hardware with far less than 512 MiB of RAM), then I'll try to test lower cased path settings.

The entry point always was 26. If I mark XUL.DLL as high (yes: again, shouldn't be a change) then the expected result after a reboot is a SYS2070 by  XUL.DLL. I'm not sure if a SYS2070 occured after marking XUL.DLL as high and after 2 reboots. If it did occur, without trying to reproduce it, then it occured once.

Quote
Also, are you trying to mark SM1.19 DLLs to load high? This is unsupported and untested and may fail
Certainly not. Only full file names specfiied by you. So just XUL.DLL in the FF/SM directories, and not yet the NS* ones which "should be safe". Code only.

Quote
Another reason to keep them loaded. BTW, why use the uni kernel? The W4 kernel will give perhaps 10% better performance and ArcaOS doesn't even include the uni kernel as it doesn't add anything but overhead.
The only reason to upgrade the default (eCS 1.2) kernel was because some software packages may require a newer version. Apparently I'm using uni20050811.zip

Quote
I've recompiled the old mozturbo with LIBPATHSTRICT if you want to test. This is only for use with SM 1.19 and is untested.
Willling. With the T42 and LIBPATHSTRICT aplies to the old  MOZTURBO.EXE Mozilla fails to start at the moment (no SYS2070), but it's the default browser of eCS 1.2. Probably 1.19; no newer or older version installed by an user.

Quote
Edit, forgot to mention that I usually run all 3 turbo programs, ffturbo, smturbo and tbturbo without problems.
Here SYS2070s after the initial SYS2070 are quite rare, there's that XWLAN issue, and I just noticed that I couldn't start Mozilla (MOZTURBO uses LIBPATHSTRICT, and so does the WPProgramobject of Mozilla itself).

Mozilla is a single, unreproduced case. The SYS2070s are too rare to not use the smart functionality, and XWLAN always has problems setting up a connected with any of the heavier *TURBOs in use. Could be reproducable with "-t" and a T42.

I've restricted the number FFTURBO.EXE and/or SMTURBO.EXE in use to 1 per install, with -l or -t. And MOZTURBO. A next test coud be to get rid of MOZTURBO, which is just an old member of an old family of products.

1. Review the existing high-installs everywhere (make take up to about one backup cycle/month).

2. Lower cased BEGINLIBPATH, with forced SYS2070 (mark XUL.DLL as high) or rare common one.

3. New MOZTURBO.EXE instead of an old one with SET LIBPATHSTRICT=... as object settings.

4. Optional: no MOZTURBO.EXE (nice-to-have as a respected member of the family, but Netscape FF/SM are users'default)

5. Mark the other DLLs "which sould be safe" as high too.

6. Try the commands "SLEEP.EXE 1 & SMTURBO.EXE -t", or something like that, to see what the effects of a delay are.

Please note that just the gain of -t is enough to justify having to set up a WLAN connect manually or having to reboot after a SYS2070 of *TURBO.EXE.

I'm expecting that the WLAN issue would be reproducable, with OS/2 or eCS software. A slower T23 (1.x GHZ Pentium III Mobile) never establishes an initial connection anymore, and so does a faster T42p Pentium 4. Especially with a T42 the eexpected result always is a succesfull WPA-connection, without any manual retry.
Title: Re: mozturbo
Post by: Pete on February 05, 2019, 05:59:21 am
Hi André

If you have XWP installed a solution to startup timing problems is to use the xwp startup folder(s) and set time delay as required to avoid startup problems.


Regards

Pete


Title: Re: mozturbo
Post by: Remy on February 05, 2019, 12:52:23 pm
Or ArcaOS startup folder 
Title: Re: mozturbo
Post by: Pete on February 05, 2019, 06:55:03 pm
Hi

Or ArcaOS startup folder


If the ArcaOS startup folder is from the cutdown XWP - that should work. I do not recall there being any setting to delay app startup in the standard OS/2 startup folder.


Regards

Pete
Title: Re: mozturbo
Post by: Dave Yeo on February 05, 2019, 08:51:07 pm
Just use startup.cmd combined with sleep().
Code: [Select]
X:
cd \path\to\Firefox
detach ffturbo
sleep 3
cd \path\to\SeaMonkey
detach smturbo
sleep 3
cd \path\to\Thunderbird
detach tbturbo
Adjust paths etc as needed. Do have to wait a few seconds during startup.
Personally, I manually start WiFi after boot on my T42.
Title: Re: mozturbo
Post by: xynixme on February 09, 2019, 02:16:18 pm
Status update: in general the all new high DLLs work everywhere (full *.DLL file names explicitly mentioned).

Since then I have experienced at least one fresh, unforced SYS2070 with SMTURBO.EXE, XUL.DLL and the same entry point.

The classic MOZTURBO (with BEGINLIBPATH) and SMTURBO.EXE (with BEGINLIBPATH) are in use.

So now I will modify (edit) the SET BEGINLIBPATH=U:\PPER\CASE to a lower case full path name. With backslashes, for all *TURBOs in my startup procedure.

I'm not going to modify the paths of all of SeaMonkey's WPProgram objects, because SMTURBO is the first entry in POPUPLOG.OS2 and manually starting SM itself will create a next entry. FWIW: there is no browser in any startup procedure.

If l:\ower\case doesn't help, then some new MOZTURBO.EXE or getting rid of the old MOZTURBO.EXE seems to be a next test.
Title: Re: mozturbo
Post by: xynixme on February 09, 2019, 02:38:31 pm
If you have XWP installed a solution to startup timing problems is to use the xwp startup folder(s) and set time delay as required to avoid startup problems.
eWP is in use. An upgrade to the light or full version is an option. A translated version is available. Albeit I first have to test that because I don't want it to screw up the eCenter look, upto and including the eCS-button people may be used to. Unfortunately there is no scheduled new install, so I haven't had an opportunity to upgrade the eWP component.

I'm not sure if a theoretical delay of less than a few milliseconds per loaded large DLL will be enough, so using STARTUP.CMD xor the virtual <WP_START> "command line /c sleep 2 & detach E:\TCPIP\SEAMONKEY\SMTURBO.EXE -t".

FWIW: of course, if needed, I'm willing to test *TURBO.EXEs with an experimental lower priority setting (DosSetPriority API). In English the typical status of XWLAN with a T42, without any delay, remains "searching connection with the current connection profile". Silently.

I will not SLEEP until the rare SYS2070s seem to be (not) solved, in order to avoid mixing tests and mixed test results.
Title: Re: mozturbo
Post by: xynixme on February 09, 2019, 02:49:35 pm
I do not recall there being any setting to delay app startup in the standard OS/2 startup folder.
No such clear setting with OS/2, eWP of eCS 1.2 and a registered DragText indeed.

I'm not using a German or English OS, so I don't know which OS introduced such a setting. eCS 2.x and/or ArcaOS. Nevertheless thank you for mentioning ArcaOS, since this is not my personal forum and you don't have to remember what I'm using. It looks like XWLAN almost gets it right, so any built-in sleep may help. Or, perhaps, a theoretical lower (TURBO.EXEs) or higher (XWLAN's connection phase) priority.

I've downloaded the SM1 MOZTURBO, but will first test the lower case full paths w.r.t. the rare SYS2070 errors (a guesstimated 5% of boots, or forceable by marking e.g. a high XUL.DLL as high).
Title: Re: mozturbo
Post by: xynixme on February 10, 2019, 01:46:57 pm
Update: still a SYS2070 (3rd entry below) with l:\ower\case BEGINLIBPATH settings, so back to U:\PPER\CASE. The 4rd entry just is the moment you are using SM for the first time and notice the 3rd entry. FWIW: PLDS4.DLL is not a mentioned full file name, so it's not marked as high. XUL.DLL of SM is.

The next test will involved your attached new MOZTURBO. Without any SLEEP, to not mix tests, and the Pentium 4 desktop computer in question has no WiFi anyway.

Code: [Select]
02-09-2019  20:19:02  SYS2070  PID 0034  TID 0001  Slot 0096
D:\SEAMONKEY\SMTURBO.EXE
XUL->PLDS4.26
182
------------------------------------------------------------

02-09-2019  21:15:26  SYS3175  PID 005f  TID 0001  Slot 00aa
D:\PROGRAMS\MOZILLA\MOZILLA.EXE
c0000005
1f4cf6cc
P1=00000001  P2=6d660001  P3=XXXXXXXX  P4=XXXXXXXX 
EAX=6d660000  EBX=022f013c  ECX=ffffffff  EDX=00000000
ESI=0012ed18  EDI=6d660001 
DS=0053  DSACC=f0f3  DSLIM=ffffffff 
ES=0053  ESACC=f0f3  ESLIM=ffffffff 
FS=150b  FSACC=00f3  FSLIM=00000030
GS=0000  GSACC=****  GSLIM=********
CS:EIP=005b:1f4cf6cc  CSACC=f0df  CSLIM=ffffffff
SS:ESP=0053:0012ecf8  SSACC=f0f3  SSLIM=ffffffff
EBP=0012ed28  FLG=00012246

LIBC066.DLL 0001:0000f6cc

------------------------------------------------------------

02-10-2019  11:24:50  SYS2070  PID 0036  TID 0001  Slot 0097
D:\SEAMONKEY\SMTURBO.EXE
XUL->PLDS4.26
182
------------------------------------------------------------

02-10-2019  13:26:31  SYS2070  PID 008b  TID 0001  Slot 009a
D:\SEAMONKEY\SEAMONKEY.EXE
XUL->PLDS4.26
182
Title: Re: mozturbo
Post by: xynixme on February 10, 2019, 03:58:00 pm
Cosmetic copy-paste mozturbo.cpp error: #define MOZ_APP_DISPLAYNAME "seamonkey.exe"
Title: Re: mozturbo
Post by: Dave Yeo on February 10, 2019, 09:09:18 pm
Cosmetic copy-paste mozturbo.cpp error: #define MOZ_APP_DISPLAYNAME "seamonkey.exe"

This is trying to compile? Weird that it would error there. Which version of GCC and what compile switches?
Title: Re: mozturbo
Post by: Dave Yeo on February 10, 2019, 09:37:08 pm
As for the sys2070's, I think this may just be an artifact of the way things work.
Smturbo preloads the DLLs in the applications directory and subdirectory, which then load their dependencies, one of which is NSPR's PLDS4, finds the mozilla.exe PLDS4.DLL in memory and uses that, possibly along with NSPR4.DLL and PLC4.DLL and loads them. These seem to be the only DLLs that have names in common between SM 1.19 (and presumably whatever version of Mozilla you're using) and SM2.49. Generally the NSPR4 stuff is very stable but obviously an extra export was added at some point and is missing from the older version. Bitwise also may have made changes that made the libs incompatible in subtle ways.
Ordering the loading so smturbo runs first and finds its proper version of NSPR etc first in LIBPATH is probably needed. Possibly it just won't work the way you're trying to use it as mozilla.exe may then try to use the newer NSPR with its modifications.
While it is possible to expand the turbo programs to take this in account, I really don't have the interest in supporting such setups.
Title: Re: mozturbo
Post by: xynixme on February 14, 2019, 02:49:17 am
Ordering the loading so smturbo runs first and finds its proper version of NSPR etc first in LIBPATH is probably needed.
LIBPATHSTRICT=T and BEGINLIBPATH=B:\Rowser's\EXE\dir were always in use with the <WP_START> turbo objects too, FWIW. I'll check the PATHs and order in use w.r.t. those shared DLL file names. If it could be a possible cause, then I'll insert such a test because even better PATHs will be a theoretical improvement anyway.

So far no rare SYS2070s with the new "seamonkey.exe"-MOZTURBO.EXE, without any SLEEP or unreported changes.  I'm waiting for the first unforced SYS2070, with this MOZTURBO.EXE

Please note I'm just reporting what's happening, without expecting support for Mensys' SM 1.19. ;)
Title: Re: mozturbo
Post by: Dave Yeo on February 14, 2019, 04:31:02 am
I guess strictly speaking, I can offer what limited support my skills enable for SM 1.19 as I built it I believe. Didn't even realize that Memsys was distributing it :)
Whether my environment could rebuild it, I'm not sure.
Title: Re: mozturbo
Post by: xynixme on February 15, 2019, 03:30:31 am
Didn't even realize that Memsys was distributing it :)
Sorry, I was probably mixing up versions & products... Too la^b^early...

Regarding the (three) *4.DLLs there shouldn't be a problem? MOZTURBO should use its own, which are located in its BEGINLIBPATH boot drive subdirectory, which isn't a LIBPATH directory (well, "." is).

With a clean boot drive policy FFTURBO xor SMTURBO should use a different copy of those *4.DLLs, which are located in a LIBPATH subdirectory.

Nevertheless I'll append that LIBPATH subdirectory to the BEGINLIBPATH too, just to make sure that it's there. Twice, and once in front.

Current <WP_START>-folder WPProgram settings (typed, errors possible), D:\INTERNET\BIN is the LIBPATH subdirectory where the modern *4.DLLs are saved:

MOZTURBO:
EXENAME=CMD.EXE
PARAMETERS=/c detach C:\PROGRAMS\MOZILLA\MOZTURBO.EXE -l
STARTUPDIR=C:\PROGRAMS\MOZILLA
SET LIBPATHSTRICT=T
SET BEGINLIBPATH=C:\PROGRAMS\MOZILLA

SMTURBO:
EXENAME=CMD.EXE
PARAMATERS=/c detach D:\SEAMONKEY\SMTURBO.EXE -t
STARTUPDIR=D:\SEAMONKEY
SET LIBPATHSTRICT=T
SET BEGINLIBPATH=D:\SEAMONKEY;D:\INTERNET\BIN

So far still no unforced SYS2070s with the "seamonkey.exe" MOZTURBO.EXE (which can be compiled with "ICC MOZTURBO.EXE" too, if I ignore the MAKE.CMD options of GGC; I may insert DosSleep() and DosSetPriority later).
Title: Re: mozturbo
Post by: Dave Yeo on February 15, 2019, 04:56:54 am
Not surprised that icc can also compile it as it is simple. Probably would have been happy without the casts I added. Seems GCC defaults to having a bunch of warnings.
You could easily recompile the seamonkey mozturbo for mozilla.exe by changing MOZ_APP_DISPLAYNAME and the other mozturbo programs should be as easy to recompile. Note I passed MOZ_APP_DISPLAYNAME on the command line. Source is at Bitbucket.
About the GCC options. -Zomf runs emxomf to convert the object file into native OS/2 obj format and uses ilink (default, wlink here) to link, also with the -g parameter, can use OS/2 debugger mostly. -Zhigh-mem causes the program to attempt to allocate memory in the high memory arena first, I had weird behaviour loading the DLLs that were marked to load high without that option, forget the specifics and it may have had a different cause.
Title: Re: mozturbo
Post by: xynixme on February 15, 2019, 05:15:05 pm
-Zhigh-mem causes the program to attempt to allocate memory in the high memory arena first, I had weird behaviour loading the DLLs that were marked to load high without that option, forget the specifics and it may have had a different cause.
1. I've seen a SYS2070 with another machine, so faulty hardware isn't a likely cause. Not that someone suggested that, but it could have been possible.

2. When I kill (C-A-D) a SM session due to a 100% CPU load (but not to PLUGIN-CONTAINER.EXE being started), then it seems like a shutdown doesn't complete anymore. CAD has to pressed to complete a regular shutdown.

3. I'm still waiting for at least one new SYS2070 to complete the current test. If a possible later code change really works for me, then I'll consider sharing the change before introducing 2 versions of a same package. I'm at least thinking about DosSetPriority tests for XWLAN (or DosSleep), assuming that such a process is slightly more time-critical than loading the DLLs. Hopefully the user won't notice a speed difference. Nothing fancy, I'm not a programmer and even a PM test version would be mainly copy & paste "programming".

No warnings with "ICC mozturbo.exe", but I didn't use that EXE. I cannot recall any IBM option related to high-memory, so hence just ICC without compiler or linker options.

I've changed the case of the paths earlier, but then a SYS2070 still occured. So it wasn't an exclusive solution. I'll lower-case it again after a new SYS2070, to avoid mixing tests now (your new NEWTURBO.EXE and longer BEGINLIBPATH for FF/SM).

0. Needless to say that the gains are outnumbering the e.g. rare SYS2070 by far. (Re-)starting the browser no longer has a project-status, for one.
Title: Re: mozturbo
Post by: xynixme on March 04, 2019, 03:06:41 am
So far no SYS2070, since Dave's new MOZTURBO.EXE... I'll continue to wait for a while, but it looks very good...
Title: Re: mozturbo
Post by: xynixme on March 14, 2019, 10:00:32 pm
Still no SYS2070s, so I'm willing to rate that as being solved by Dave's new MOZTURBO.EXE. Soon I'll try a few modifications, like DosSetPriority or DosSleep to try to avoid the issue of significantly less initial Wi-Fi connections by XWLAN, and perhaps an unwanted test move to PM to make regular shutdowns work more often.
Title: Re: mozturbo
Post by: Neil Waldhauer on March 24, 2019, 01:49:12 am
I put both ffturbo and tbturbo in my startup folder. Usually no problem. Just now, I saw this happen at startup.

Code: [Select]
03-23-2019  17:42:47  SYS3175  PID 004a  TID 0001  Slot 008a
K:\PROGRAMS\THUNDERBIRD\TBTURBO.EXE
c0000005
1fb7ec7a
P1=00000001  P2=1df90096  P3=XXXXXXXX  P4=XXXXXXXX 
EAX=00000000  EBX=0000c000  ECX=1df90014  EDX=00000000
ESI=1a2f5e60  EDI=18f7072a 
DS=0053  DSACC=d0f3  DSLIM=5fffffff 
ES=0053  ESACC=d0f3  ESLIM=5fffffff 
FS=150b  FSACC=00f3  FSLIM=00000030
GS=dddf  GSACC=00f3  GSLIM=000005cf
CS:EIP=005b:1fb7ec7a  CSACC=d0df  CSLIM=5fffffff
SS:ESP=0053:0015e6d0  SSACC=d0f3  SSLIM=5fffffff
EBP=0015ec68  FLG=00010246

PMMERGE.DLL 0004:000aec7a

Code: [Select]
{0}[k:\programs\thunderbird] dir tbturbo.exe

 Volume in drive K is ArcaOS-K       Serial number is 0712:95BD
 Directory of  K:\programs\Thunderbird\tbturbo.exe

12-24-18  19:58           8,304    124  tbturbo.exe
           8,304 bytes in 1 file and 0 dirs
          12,288 bytes allocated
   1,369,518,080 bytes (1,306MB) free
Title: Re: mozturbo
Post by: Lars on March 24, 2019, 07:10:58 pm
I looked at the tbturbo/ffturbo code: it lists "mozalloc.dll" as a file to preload. But I cannot find that file, neither in my firefox nor in my Thunderbird directory.
I remember that it showed up there in the past but no longer (maybe I deleted it as I identified it as superfluous).
Maybe that file should be removed from the list of files to load.
Neil, if you still find this file in your FF or TB directories, try to remove it or rename it and see if that helps.
Title: Re: mozturbo
Post by: Dave Yeo on March 24, 2019, 08:05:24 pm
The mozalloc.dll is in there for 38ESR and as far as I understand the code, DosLoadModule() will just fail when trying to load it and move on to the next DLL in the list.
Title: Re: mozturbo
Post by: Neil Waldhauer on March 25, 2019, 02:53:06 am
I only see mozalloc.dll in the Firefox 38 directory. It shouldn't have loaded. And normally there is no error. There was just the one time, I noticed the tbturbo had not started and a looked in POPUPLOG.OS2 and saw an error at the same time.
Title: Re: mozturbo
Post by: Dave Yeo on March 25, 2019, 04:48:46 am
Yours is the only report (besides the sys2070's up the page) of a sys3175 or other crash. I'm not good at reading the popup but at a guess a rare race condition with both starting. Personally I have all 3 in startup folder and haven't had any problems.