Author Topic: mozturbo  (Read 4390 times)

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 10
  • -Receive: 139
  • Posts: 1987
    • View Profile
Re: mozturbo
« Reply #15 on: December 27, 2018, 06:47:22 pm »
Hi Dariusz. The difference is in how much of the DLLs get preloaded. With the new setting, the DLLs just have a smaller memory footprint. You can test it yourself using Theseus. Choose a process, any process seems to work as we're looking at shared memory, and then choose Process-->Shared Object Summary and scroll down towards the bottom of the new window where high memory is and observe how much memory xul.dll for example is using.
As for brwscmp.dll, I missed that with the first release, which I just deleted. Don't use that release with Firefox as the missing brwscmp.dll caused buggy behaviour with Firefoxes memory. SM and TB were fine with that release.
Clearkey.dll shouldn't even be installed as it does nothing on our platform currently and probably never will. It's for DRM managing H264 files, things like Netflix I guess. See https://wiki.mozilla.org/GeckoMediaPlugins. The fact it is never loaded can be verified with Theseus or simply by moving it out of the way with Firefox running. Even if it got dynamically loaded like avcodec (FFmpeg), unless it is marked to load high, it wouldn't particularly matter.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Thank You
  • -Given: 13
  • -Receive: 9
  • Posts: 534
    • View Profile
Re: mozturbo
« Reply #16 on: December 28, 2018, 02:15:37 am »
...As for brwscmp.dll, I missed that with the first release, which I just deleted. Don't use that release with Firefox as the missing brwscmp.dll caused buggy behaviour with Firefoxes memory. SM and TB were fine with that release...

Ahhh...indeed, I remember one of your emails on our tester list about this, sorry to bring that one back, I am using your latest release now and was previously using the fixed one which picked that DLL as well.

Alright, I will take a look at the default and the '-t' optional switch and will post the results I'm seeing on my machine.

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 45
  • Posts: 1175
    • View Profile
Re: mozturbo
« Reply #17 on: January 04, 2019, 06:11:13 am »
Quote
Updated to v0.5. Default is now not to force load the modules, saving quite a bit of memory at the cost of turbo functionality. Thanks to Rich for pointing this out. Added the -t parameter for the old behaviour.
I don't anticipate any more binary changes to the programs.

I just want to report, that the Firefox version is working well (no parameters). I had to reboot, for a different problem, today, after 7 days up time (first time, in a long time). Something is still eating lower shared memory, but it seems to taper off, after a few days.

Thank you.

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 10
  • -Receive: 139
  • Posts: 1987
    • View Profile
Re: mozturbo
« Reply #18 on: January 05, 2019, 01:02:18 am »
Good to hear. I was on the 8th day of uptime before the wind storm took out the power. That was with restarting mozturbo a few times for testing purposes. It seems like it has been decades since I got that much uptime.
Be interesting to know where you're lower memory is going. Looking here, the DLLs still use some lower memory, XUL and such grab some private memory in the shared arena for some reason and of course all the supporting DLLs load in low memory and seem to use some for private purposes as well.
Then there are things like FC/2, which when I started it up, used quite a bit of low memory, 30 odd MBs. Looking, it all seemed to be claimed by PMMERGE.

I also marked some of the supporting DLLs to load high, namely icudt.dll, icuin.dll, icuuc.dll, hunspell10.dll, libvpx4.dll as they used the most and seemed safe. The NSPR and NSS DLLs are probably safe as well. Others, I'm reluctant to mark high as I don't think all programs can use them if marked high. I marked Cairo at one point and got weird results from the screensaver and even mozturbo gave me weird results until I added --Zhigh-mem to the LDFLAGS so have to be careful what is marked high.
They all need unlocking before marking as well as mozturbo loads them all and a restart of mozturbo or reboot is required to actually use them.
Have to remember that if any of these DLLs are upgraded, they'll also be locked.

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 45
  • Posts: 1175
    • View Profile
Re: mozturbo
« Reply #19 on: January 05, 2019, 07:24:05 am »
Quote
Be interesting to know where you're lower memory is going. Looking here, the DLLs still use some lower memory, XUL and such grab some private memory in the shared arena for some reason and of course all the supporting DLLs load in low memory and seem to use some for private purposes as well.
Then there are things like FC/2, which when I started it up, used quite a bit of low memory, 30 odd MBs. Looking, it all seemed to be claimed by PMMERGE.

I haven't taken the time to try to figure out where the lower shared memory is going. Upper shared memory is also decreasing, but that isn't as serious, because it ends up around 1 GB left, while lower shared memory goes as low as 48 meg (when it continues to run). It seems that 100 meg of shared memory can disappear very quickly, sometimes. The main user, that I know about, is VBOX 5.0.6, which uses about 100 meg of lower shared memory. Once it gets itself running, it doesn't use any more, and it seems to return all of it when it terminates. The problem is, that other programs may need lower shared memory, while VBOX is running, and that can cause problems. I did try adjusting the VBOX DLLs to load high, but that didn't work, and I haven't tried again (yet).

Quote
I also marked some of the supporting DLLs to load high, namely icudt.dll, icuin.dll, icuuc.dll, hunspell10.dll, libvpx4.dll as they used the most and seemed safe. The NSPR and NSS DLLs are probably safe as well. Others, I'm reluctant to mark high as I don't think all programs can use them if marked high. I marked Cairo at one point and got weird results from the screensaver and even mozturbo gave me weird results until I added --Zhigh-mem to the LDFLAGS so have to be careful what is marked high.

Common DLLs, used by many, different, programs are probably not going to work well, if marked high. Anything used by only a single program (or perhaps a suite of related programs) is probably okay, as long as all of the programs can handle it. The main problem is, that there could be one thing, that isn't used much, that will fail. By the time that happens, the user will have forgotten what they did, and the relationship may not be obvious.

Oh well, it is still a step in the right direction, until somebody can figure out how to fix the problem properly.

Quote
They all need unlocking before marking as well as mozturbo loads them all and a restart of mozturbo or reboot is required to actually use them.
Have to remember that if any of these DLLs are upgraded, they'll also be locked.

Unloading MozTurbo would seem to be asking for trouble, since the main purpose is to keep the DLLs loaded, to prevent the DLL unload problem. I expect that any future updates would be done with RPM/YUM, and that seems to handle locked files okay. A reboot is probably the best way to handle that, but I am not sure how the user could be told to do a reboot (which is probably a good idea after updating those things anyway).

André Heldoorn

  • Hero Member
  • *****
  • Thank You
  • -Given: 62
  • -Receive: 16
  • Posts: 562
    • View Profile
Re: mozturbo
« Reply #20 on: January 05, 2019, 10:28:51 am »
SMTURBO.EXE -T wasn't running anymore after several failing attempts to start SEAMONKEY.EXE, each resulting in a SYS2070. I was able to start SEAMONKEY.EXE after killing MOZTURBO.EXE -L as well, but that may be unrelated. SEAMONKEY.EXE was the first started, related product after a cold boot (loading was finished, I was editing a file before opening the browser).

01-05-2019  10:16:05  SYS2070  PID 0039  TID 0001  Slot 009a
D:\SEAMONKEY\SEAMONKEY.EXE
XUL->PLDS4.26
182

FWIW.

André Heldoorn

  • Hero Member
  • *****
  • Thank You
  • -Given: 62
  • -Receive: 16
  • Posts: 562
    • View Profile
Re: mozturbo
« Reply #21 on: January 05, 2019, 11:17:42 am »
The main user, that I know about, is VBOX 5.0.6, which uses about 100 meg of lower shared memory.
FWIW: Object Rexx' file-I/O, otherwise a better Rexx interpreter, is a good way to run out of shared memory too. Until a WPS reset. I've never checked if ORexx would beat ~100 MiB, though.

André Heldoorn

  • Hero Member
  • *****
  • Thank You
  • -Given: 62
  • -Receive: 16
  • Posts: 562
    • View Profile
Re: mozturbo
« Reply #22 on: January 05, 2019, 11:34:06 am »
using the FCF_TASKLIST flag means a full PM app, I believe it applies to the frame, something that would be overkill
A hidden frame, FWIW. So basicly the same effects as Rich's DETACH, regardless of the abuse of democracy by claiming "most of us are using ...": not visible, no shutdown dialogs, no requirements of requirements to avoid useless dialogs. ;-)

Which slower hardware and DETACH it'll appear in whatever task list a few people are using for a few zilliseconds, but that's it. It may be possible to hide selected apps, but that's not required with solutions like (not) FCF_TASKLIST or DETACH. DETACH is better, since there is no meaningful PM UI.

I've never tried to replace/upgrade eCenter bij xCenter, so apparently I'm not using what "most of us are using" indeed. That rather popular assumption is right. :-)

André Heldoorn

  • Hero Member
  • *****
  • Thank You
  • -Given: 62
  • -Receive: 16
  • Posts: 562
    • View Profile
Re: mozturbo
« Reply #23 on: January 05, 2019, 11:59:06 am »
I also marked some of the supporting DLLs to load high, namely icudt.dll, icuin.dll, icuuc.dll, hunspell10.dll, libvpx4.dll as they used the most and seemed safe.
Regarding HUNSPEL0.DLL (8.3 DLL file name): I've just installed this DLL and no other files. So mine doesn't load and expand any dictionary files or other files, if that's what HUNSPEL0.DLL would be doing with your RAM. As such FF/SM only requires HUNSPEL0.DLL of that package, which doesn't appear to be a large file.

Nevertheless I'll mark HUNSPEL0.DLL as high, since it seems to be safe and I'm pretty sure no other app uses it. It just has to be there, as some useless technical requirement.

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 45
  • Posts: 1175
    • View Profile
Re: mozturbo
« Reply #24 on: January 05, 2019, 06:06:58 pm »
Quote
FWIW: Object Rexx' file-I/O, otherwise a better Rexx interpreter, is a good way to run out of shared memory too. Until a WPS reset. I've never checked if ORexx would beat ~100 MiB, though.

OREXX is known to have some serious problems. It is recommended to avoid using it, when possible.

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 10
  • -Receive: 139
  • Posts: 1987
    • View Profile
Re: mozturbo
« Reply #25 on: January 05, 2019, 06:28:24 pm »
SMTURBO.EXE -T wasn't running anymore after several failing attempts to start SEAMONKEY.EXE, each resulting in a SYS2070. I was able to start SEAMONKEY.EXE after killing MOZTURBO.EXE -L as well, but that may be unrelated. SEAMONKEY.EXE was the first started, related product after a cold boot (loading was finished, I was editing a file before opening the browser).

01-05-2019  10:16:05  SYS2070  PID 0039  TID 0001  Slot 009a
D:\SEAMONKEY\SEAMONKEY.EXE
XUL->PLDS4.26
182

FWIW.

Interesting, it is a NSPR DLL. Do you have something running that also uses NSPR? Old version of a Mozilla App? Seems to me other stuff has used NSPR but can't think of any right now.
Mozturbo will load the other DLLs required by SeaMonkey etc and here it must have been loading the wrong one.
BTW, SeaMonkey doesn't use the Hunspell package but rather the one included with Mozilla so no need for it.

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 10
  • -Receive: 139
  • Posts: 1987
    • View Profile
Re: mozturbo
« Reply #26 on: January 05, 2019, 07:22:12 pm »
Hi Doug, unloading and reloading mozturbo a couple of times will just waste some address space. While better to avoid, it didn't hurt here besides having available upper address space shrink.
Seems that for vbox, it is the QT DLLs eating up lower address space and leaving some fragmentation when closed and perhaps a leak as on one quick test, pmmerge seems to be left allocating quite a bit of low memory.
This raises the question of whether the QT4 DLLs can be safely loaded high and then we'll also have the same question about the QT5 DLLs. In this 64bit world, memory usage of ported apps is likely to increase. One thing is that it would be easy to modify mozturbo into qtturbo.

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 45
  • Posts: 1175
    • View Profile
Re: mozturbo
« Reply #27 on: January 05, 2019, 08:10:55 pm »
Quote
Hi Doug, unloading and reloading mozturbo a couple of times will just waste some address space. While better to avoid, it didn't hurt here besides having available upper address space shrink.

I found that loading/unloading Firefox, while using the DLLs in high memory, worked, most of the time, but not always. Throw in the leakage, and it didn't take very many load/unload cycles to cause problems. MozTurbo seems to work around the worst of the bad things that Firefox was doing. Unfortunately, other things need the same treatment, and that could go on forever. Many things also need to be rebuilt, to accept loading DLLs high. All we can do, is find programs that are capable, and make sure that they load high, and don't unload DLLs (at least until the problem gets fixed, which may be never).

Quote
This raises the question of whether the QT4 DLLs can be safely loaded high and then we'll also have the same question about the QT5 DLLs.

QT is used by a number of programs. If ALL of them are capable, it should be okay. If not, it won't be okay. VBox seems to be well written, and it does return all of the used shared memory (according to Above512, which isn't really accurate). The big question, that I have, is why is it using so much shared memory? I am pretty sure that it isn't all DLLs, and at least some of it could probably be loaded high.

Lots of experimenting to do.  :)

David McKenna

  • Full Member
  • ***
  • Thank You
  • -Given: 8
  • -Receive: 9
  • Posts: 200
    • View Profile
Re: mozturbo
« Reply #28 on: January 05, 2019, 08:55:16 pm »
Hi Dave -

  Thanks for the SMTurbo thing - it seems to have made my system much more stable in general, and like others have said, uptime for SeaMonkey is much longer.

  Just thinking out loud... would it be possible to alter SMTurbo into a more generic 'Turbo' program that uses a text file containing a list of DLL's that can be loaded at startup (instead of hard coded)? Might be helpful with other sub-system DLL's that get loaded/unloaded over time...

Regards,
« Last Edit: January 05, 2019, 09:04:25 pm by David McKenna »

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 10
  • -Receive: 139
  • Posts: 1987
    • View Profile
Re: mozturbo
« Reply #29 on: January 05, 2019, 09:25:38 pm »
  Just thinking out loud... would it be possible to alter SMTurbo into a more generic 'Turbo' program that uses a text file containing a list of DLL's that can be loaded at startup (instead of hard coded)? Might be helpful with other sub-system DLL's that get loaded/unloaded over time...

I don't see why not, especially if the DLLs are on LIBPATH (and perhaps BEGINLIBPATH if wrapped in a script).
I'll give it some thought.