Author Topic: Curious what the problem is making new firefox versions.  (Read 2287 times)

David Kiley

  • Full Member
  • ***
  • Thank You
  • -Given: 9
  • -Receive: 2
  • Posts: 130
  • Karma: +2/-0
    • View Profile
Curious what the problem is making new firefox versions.
« on: December 19, 2020, 07:20:26 am »
I'm not a programmer so i'm curious to understand why making new mozilla builds is troublesome?

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 13
  • -Receive: 225
  • Posts: 3140
  • Karma: +57/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #1 on: December 19, 2020, 08:03:57 am »
I'm not a programmer either but learning :) I do have the source for 52ESR here that almost compiles. After 52ESR, Mozilla switched to using the programming language Rust to build various modules. We have no Rust port, a very big job and it seems that linking with Rust modules needs much more memory then a 32 bit platform has. Even by 45ESR (actually it was worse for a while), linking xul.dll basically requires the full possible address range. Back when I only had 1.5 GBs of ram for a while, I actually crashed the machine due to the swap file filling up. Seems it has a 2GB limit, which I'm sure seemed like a lot in the early 90's.
It's a problem with the new browser as well, along with compiling some of the object files eating tons of memory. Dmik had to cut down the number of jobs for make to stop from running out of memory.
Theoretically, it should be possible to cross compile OS/2 binaries on Linux or even Windows, but once again it would be a big job by a knowledgeable person setting things up.
Eventually this will kill OS/2 as developers use 64bit machines full of memory, less and less will run on 32 bit machines.

David Kiley

  • Full Member
  • ***
  • Thank You
  • -Given: 9
  • -Receive: 2
  • Posts: 130
  • Karma: +2/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #2 on: December 19, 2020, 10:48:21 am »
Eventually this will kill OS/2 as developers use 64bit machines full of memory, less and less will run on 32 bit machines.
Thanks for the explanation - I didn't know it was so connected to the ram requirements. Fascinating and sad if it leads to the death of the OS. How does the QT browser solve that problem though?

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 13
  • -Receive: 225
  • Posts: 3140
  • Karma: +57/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #3 on: December 19, 2020, 04:56:47 pm »
How does the QT browser solve that problem though?
[/quote]

It is currently barely capable of being built, carefully.

JTC

  • Newbie
  • *
  • Thank You
  • -Given: 0
  • -Receive: 0
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #4 on: December 19, 2020, 08:53:46 pm »
Are there any compile-time optimizations that could help further reduce the size of the binaries ?  Or compile options to target multi-core chips with instructions that
can further reduce the execution path ??

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Thank You
  • -Given: 607
  • -Receive: 122
  • Posts: 3296
  • Karma: +27/-0
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: Curious what the problem is making new firefox versions.
« Reply #5 on: December 19, 2020, 09:04:19 pm »
FYI: There was some past discussion here - "Sponsorship needed for new OS/2 web browser  (2017-11-17)".  Wow... it was three years ago.

Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 13
  • -Receive: 225
  • Posts: 3140
  • Karma: +57/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #6 on: December 20, 2020, 01:06:32 am »
Are there any compile-time optimizations that could help further reduce the size of the binaries ?  Or compile options to target multi-core chips with instructions that
can further reduce the execution path ??

-Os for a smaller optimized binary. Experimenting with Firefox (I believe there is a copy at Bitbucket) it wasn't that much smaller and slightly slower.
The thing with building on a multicore machine is that make can do more jobs by forking. I usually use -j5 (-j3 seems the most that works for TB) so that at times there are 5 make.exe's compiling different subdirectories on my 4 core machine, the extra one is for when one is blocked waiting for the HD. Nice to have all 4 cores running at 100% much of the time. Of course there are bottlenecks where one or more make's has to wait for a dependency to finish.
Bitwise was running into the situation where some object files were using 2GB to compile so had to revert to -j2 or -j1 to avoid the system hanging, leading to long compile times.

The compiler doesn't really seem to have options for multi-core. I did try tuning my latest for Core2 but whether it made a difference is hard to tell. I believe there are libraries that help for multi-core but basically just using more threads is key. While Mozilla uses threads, most of it runs on thread 1.
There's also having each tab as its own process, an option with QT and Firefox. Firefox is broken for doing it in our port and there are memory problems again with having a process per tab besides harder to make work (they have to communicate). Advantage is having a single tab crash instead of the whole browser. I expect that the QT browser will have that disabled, especially at first.
« Last Edit: December 20, 2020, 01:08:40 am by Dave Yeo »

Roderick Klein

  • Sr. Member
  • ****
  • Thank You
  • -Given: 0
  • -Receive: 26
  • Posts: 426
  • Karma: +6/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #7 on: December 20, 2020, 02:12:04 am »
FYI: There was some past discussion here - "Sponsorship needed for new OS/2 web browser  (2017-11-17)".  Wow... it was three years ago.

Regards

It took also much longer as the code is more modern.
It required work on GCC 9.2 compiler and custom code in LIBCX and a lot of other DLL's added/updated .

Roderick

David Kiley

  • Full Member
  • ***
  • Thank You
  • -Given: 9
  • -Receive: 2
  • Posts: 130
  • Karma: +2/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #8 on: December 20, 2020, 07:05:49 am »
It is currently barely capable of being built, carefully.

Yeah I know it's still under development just curious if it would solve the memory issue if/when it gets functional or that is just something fundamental.

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 13
  • -Receive: 225
  • Posts: 3140
  • Karma: +57/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #9 on: December 20, 2020, 09:15:06 am »
It is currently barely capable of being built, carefully.

Yeah I know it's still under development just curious if it would solve the memory issue if/when it gets functional or that is just something fundamental.

Well browsers are just using more and more memory. I'd assume that it'll keep working for quite a while, especially if the users are careful not to have a hundred tabs open but fundamentally memory usage keeps increasing and we have a limited amount.
It's weird, I started using OS/2 v3 on a 4MB machine and was so happy when I got 8 MB's. Now with 12 GB's of memory, I worry more about memory. Before it was excessive swapping, now it is crashing.

Neil Waldhauer

  • Hero Member
  • *****
  • Thank You
  • -Given: 14
  • -Receive: 62
  • Posts: 742
  • Karma: +20/-0
    • View Profile
    • Blonde Guy
Re: Curious what the problem is making new firefox versions.
« Reply #10 on: December 20, 2020, 04:23:37 pm »
I may be wandering off-topic, but if the runtime libraries supporting Chromium (or even Firefox) fail gracefully when running out of memory and the applications behave well when receiving an out of resources error, we might function pretty well in the future.
Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Roderick Klein

  • Sr. Member
  • ****
  • Thank You
  • -Given: 0
  • -Receive: 26
  • Posts: 426
  • Karma: +6/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #11 on: December 20, 2020, 06:56:13 pm »
Are there any compile-time optimizations that could help further reduce the size of the binaries ?  Or compile options to target multi-core chips with instructions that
can further reduce the execution path ??

-Os for a smaller optimized binary. Experimenting with Firefox (I believe there is a copy at Bitbucket) it wasn't that much smaller and slightly slower.
The thing with building on a multicore machine is that make can do more jobs by forking. I usually use -j5 (-j3 seems the most that works for TB) so that at times there are 5 make.exe's compiling different subdirectories on my 4 core machine, the extra one is for when one is blocked waiting for the HD. Nice to have all 4 cores running at 100% much of the time. Of course there are bottlenecks where one or more make's has to wait for a dependency to finish.
Bitwise was running into the situation where some object files were using 2GB to compile so had to revert to -j2 or -j1 to avoid the system hanging, leading to long compile times.

The compiler doesn't really seem to have options for multi-core. I did try tuning my latest for Core2 but whether it made a difference is hard to tell. I believe there are libraries that help for multi-core but basically just using more threads is key. While Mozilla uses threads, most of it runs on thread 1.
There's also having each tab as its own process, an option with QT and Firefox. Firefox is broken for doing it in our port and there are memory problems again with having a process per tab besides harder to make work (they have to communicate). Advantage is having a single tab crash instead of the whole browser. I expect that the QT browser will have that disabled, especially at first.

That last bit is being worked so when a single tab crashes it does not take down the complete browser.

As for the compiler issue. I I get it correctly the following issue's exists:

1) We can not run multiple instance of GCC.EXE parallel to compile the webkit DLL.

2) By being more selective with large DLL which debug information you compile you can reduce the memory needed.

3) To reduce compile time and memory usage tweaks are possible on WLINK and GCC. But it takes time.

Roderick

David Graser

  • Hero Member
  • *****
  • Thank You
  • -Given: 0
  • -Receive: 89
  • Posts: 680
  • Karma: +43/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #12 on: December 20, 2020, 07:02:40 pm »
I am not a programmer. 

I have always wondered why RAM could not be compressed like the OS/2 program Zipstream, a disk compression utility used back in the nineties when hard drives were small?  Compress a RAM drive where you 6 MB of RAM down to 4 GB of RAM. This would allow one to create a bigger RAM drive.  Since the drive is temporary, more RAM could be used for the drive.  Compress RAM and would that not allow one to use more physical RAM?

http://os2ezine.com/v2n1/arc.htm

Roderick Klein

  • Sr. Member
  • ****
  • Thank You
  • -Given: 0
  • -Receive: 26
  • Posts: 426
  • Karma: +6/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #13 on: December 20, 2020, 07:05:28 pm »
Eventually this will kill OS/2 as developers use 64bit machines full of memory, less and less will run on 32 bit machines.
Thanks for the explanation - I didn't know it was so connected to the ram requirements. Fascinating and sad if it leads to the death of the OS. How does the QT browser solve that problem though?

While its true I do see options with R&D making it possible to last longer.

As I mentioned tweaks can be made to GCC compiler on ArcaOS and the wlink.exe can be modified. Also by being selective on the amount of debug information that gets added while compiling you should also take into account.

From what I can tell compiling QT for Bitwise Works was not an issue. It was because of the webkit DLL that is so big.
The DLL is 250 MB and the other QT 5 DLL's are a lot smaller.

Roderick

David Kiley

  • Full Member
  • ***
  • Thank You
  • -Given: 9
  • -Receive: 2
  • Posts: 130
  • Karma: +2/-0
    • View Profile
Re: Curious what the problem is making new firefox versions.
« Reply #14 on: December 25, 2020, 02:07:36 am »
From what I can tell compiling QT for Bitwise Works was not an issue. It was because of the webkit DLL that is so big.
The DLL is 250 MB and the other QT 5 DLL's are a lot smaller.
Why does the size of the file matter, if it doesn't need more memory to compile?

Also, hair-brained idea:
Could you take an open source compiler from say freedos or reactos, and then add the function to use a swap file from a ramdisk, to reach higher memory when compiling?
Keep in mind a have no idea what i'm talking about really, just blue skying.
I just remember in the DOS days I used a compiler from borland c++ that would generate .exe files.. so it made me think maybe there is some open source compiler that would work with OS/2, from one of those other projects.