OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Internet => Topic started by: David Kiley on December 19, 2020, 07:20:26 am

Title: Curious what the problem is making new firefox versions.
Post by: David Kiley on December 19, 2020, 07:20:26 am
I'm not a programmer so i'm curious to understand why making new mozilla builds is troublesome?
Title: Re: Curious what the problem is making new firefox versions.
Post by: Dave Yeo on December 19, 2020, 08:03:57 am
I'm not a programmer either but learning :) I do have the source for 52ESR here that almost compiles. After 52ESR, Mozilla switched to using the programming language Rust to build various modules. We have no Rust port, a very big job and it seems that linking with Rust modules needs much more memory then a 32 bit platform has. Even by 45ESR (actually it was worse for a while), linking xul.dll basically requires the full possible address range. Back when I only had 1.5 GBs of ram for a while, I actually crashed the machine due to the swap file filling up. Seems it has a 2GB limit, which I'm sure seemed like a lot in the early 90's.
It's a problem with the new browser as well, along with compiling some of the object files eating tons of memory. Dmik had to cut down the number of jobs for make to stop from running out of memory.
Theoretically, it should be possible to cross compile OS/2 binaries on Linux or even Windows, but once again it would be a big job by a knowledgeable person setting things up.
Eventually this will kill OS/2 as developers use 64bit machines full of memory, less and less will run on 32 bit machines.
Title: Re: Curious what the problem is making new firefox versions.
Post by: David Kiley on December 19, 2020, 10:48:21 am
Eventually this will kill OS/2 as developers use 64bit machines full of memory, less and less will run on 32 bit machines.
Thanks for the explanation - I didn't know it was so connected to the ram requirements. Fascinating and sad if it leads to the death of the OS. How does the QT browser solve that problem though?
Title: Re: Curious what the problem is making new firefox versions.
Post by: Dave Yeo on December 19, 2020, 04:56:47 pm
How does the QT browser solve that problem though?
[/quote]

It is currently barely capable of being built, carefully.
Title: Re: Curious what the problem is making new firefox versions.
Post by: JTCaptKirk on December 19, 2020, 08:53:46 pm
Are there any compile-time optimizations that could help further reduce the size of the binaries ?  Or compile options to target multi-core chips with instructions that
can further reduce the execution path ??
Title: Re: Curious what the problem is making new firefox versions.
Post by: Martin Iturbide on December 19, 2020, 09:04:19 pm
FYI: There was some past discussion here - "Sponsorship needed for new OS/2 web browser  (2017-11-17)" (https://www.os2world.com/forum/index.php/topic,1491.msg14934.html#msg14934).  Wow... it was three years ago.

Regards
Title: Re: Curious what the problem is making new firefox versions.
Post by: Dave Yeo on December 20, 2020, 01:06:32 am
Are there any compile-time optimizations that could help further reduce the size of the binaries ?  Or compile options to target multi-core chips with instructions that
can further reduce the execution path ??

-Os for a smaller optimized binary. Experimenting with Firefox (I believe there is a copy at Bitbucket) it wasn't that much smaller and slightly slower.
The thing with building on a multicore machine is that make can do more jobs by forking. I usually use -j5 (-j3 seems the most that works for TB) so that at times there are 5 make.exe's compiling different subdirectories on my 4 core machine, the extra one is for when one is blocked waiting for the HD. Nice to have all 4 cores running at 100% much of the time. Of course there are bottlenecks where one or more make's has to wait for a dependency to finish.
Bitwise was running into the situation where some object files were using 2GB to compile so had to revert to -j2 or -j1 to avoid the system hanging, leading to long compile times.

The compiler doesn't really seem to have options for multi-core. I did try tuning my latest for Core2 but whether it made a difference is hard to tell. I believe there are libraries that help for multi-core but basically just using more threads is key. While Mozilla uses threads, most of it runs on thread 1.
There's also having each tab as its own process, an option with QT and Firefox. Firefox is broken for doing it in our port and there are memory problems again with having a process per tab besides harder to make work (they have to communicate). Advantage is having a single tab crash instead of the whole browser. I expect that the QT browser will have that disabled, especially at first.
Title: Re: Curious what the problem is making new firefox versions.
Post by: Roderick Klein on December 20, 2020, 02:12:04 am
FYI: There was some past discussion here - "Sponsorship needed for new OS/2 web browser  (2017-11-17)" (https://www.os2world.com/forum/index.php/topic,1491.msg14934.html#msg14934).  Wow... it was three years ago.

Regards

It took also much longer as the code is more modern.
It required work on GCC 9.2 compiler and custom code in LIBCX and a lot of other DLL's added/updated .

Roderick
Title: Re: Curious what the problem is making new firefox versions.
Post by: David Kiley on December 20, 2020, 07:05:49 am
It is currently barely capable of being built, carefully.

Yeah I know it's still under development just curious if it would solve the memory issue if/when it gets functional or that is just something fundamental.
Title: Re: Curious what the problem is making new firefox versions.
Post by: Dave Yeo on December 20, 2020, 09:15:06 am
It is currently barely capable of being built, carefully.

Yeah I know it's still under development just curious if it would solve the memory issue if/when it gets functional or that is just something fundamental.

Well browsers are just using more and more memory. I'd assume that it'll keep working for quite a while, especially if the users are careful not to have a hundred tabs open but fundamentally memory usage keeps increasing and we have a limited amount.
It's weird, I started using OS/2 v3 on a 4MB machine and was so happy when I got 8 MB's. Now with 12 GB's of memory, I worry more about memory. Before it was excessive swapping, now it is crashing.
Title: Re: Curious what the problem is making new firefox versions.
Post by: Neil Waldhauer on December 20, 2020, 04:23:37 pm
I may be wandering off-topic, but if the runtime libraries supporting Chromium (or even Firefox) fail gracefully when running out of memory and the applications behave well when receiving an out of resources error, we might function pretty well in the future.
Title: Re: Curious what the problem is making new firefox versions.
Post by: Roderick Klein on December 20, 2020, 06:56:13 pm
Are there any compile-time optimizations that could help further reduce the size of the binaries ?  Or compile options to target multi-core chips with instructions that
can further reduce the execution path ??

-Os for a smaller optimized binary. Experimenting with Firefox (I believe there is a copy at Bitbucket) it wasn't that much smaller and slightly slower.
The thing with building on a multicore machine is that make can do more jobs by forking. I usually use -j5 (-j3 seems the most that works for TB) so that at times there are 5 make.exe's compiling different subdirectories on my 4 core machine, the extra one is for when one is blocked waiting for the HD. Nice to have all 4 cores running at 100% much of the time. Of course there are bottlenecks where one or more make's has to wait for a dependency to finish.
Bitwise was running into the situation where some object files were using 2GB to compile so had to revert to -j2 or -j1 to avoid the system hanging, leading to long compile times.

The compiler doesn't really seem to have options for multi-core. I did try tuning my latest for Core2 but whether it made a difference is hard to tell. I believe there are libraries that help for multi-core but basically just using more threads is key. While Mozilla uses threads, most of it runs on thread 1.
There's also having each tab as its own process, an option with QT and Firefox. Firefox is broken for doing it in our port and there are memory problems again with having a process per tab besides harder to make work (they have to communicate). Advantage is having a single tab crash instead of the whole browser. I expect that the QT browser will have that disabled, especially at first.

That last bit is being worked so when a single tab crashes it does not take down the complete browser.

As for the compiler issue. I I get it correctly the following issue's exists:

1) We can not run multiple instance of GCC.EXE parallel to compile the webkit DLL.

2) By being more selective with large DLL which debug information you compile you can reduce the memory needed.

3) To reduce compile time and memory usage tweaks are possible on WLINK and GCC. But it takes time.

Roderick
Title: Re: Curious what the problem is making new firefox versions.
Post by: David Graser on December 20, 2020, 07:02:40 pm
I am not a programmer. 

I have always wondered why RAM could not be compressed like the OS/2 program Zipstream, a disk compression utility used back in the nineties when hard drives were small?  Compress a RAM drive where you 6 MB of RAM down to 4 GB of RAM. This would allow one to create a bigger RAM drive.  Since the drive is temporary, more RAM could be used for the drive.  Compress RAM and would that not allow one to use more physical RAM?

http://os2ezine.com/v2n1/arc.htm
Title: Re: Curious what the problem is making new firefox versions.
Post by: Roderick Klein on December 20, 2020, 07:05:28 pm
Eventually this will kill OS/2 as developers use 64bit machines full of memory, less and less will run on 32 bit machines.
Thanks for the explanation - I didn't know it was so connected to the ram requirements. Fascinating and sad if it leads to the death of the OS. How does the QT browser solve that problem though?

While its true I do see options with R&D making it possible to last longer.

As I mentioned tweaks can be made to GCC compiler on ArcaOS and the wlink.exe can be modified. Also by being selective on the amount of debug information that gets added while compiling you should also take into account.

From what I can tell compiling QT for Bitwise Works was not an issue. It was because of the webkit DLL that is so big.
The DLL is 250 MB and the other QT 5 DLL's are a lot smaller.

Roderick
Title: Re: Curious what the problem is making new firefox versions.
Post by: David Kiley on December 25, 2020, 02:07:36 am
From what I can tell compiling QT for Bitwise Works was not an issue. It was because of the webkit DLL that is so big.
The DLL is 250 MB and the other QT 5 DLL's are a lot smaller.
Why does the size of the file matter, if it doesn't need more memory to compile?

Also, hair-brained idea:
Could you take an open source compiler from say freedos or reactos, and then add the function to use a swap file from a ramdisk, to reach higher memory when compiling?
Keep in mind a have no idea what i'm talking about really, just blue skying.
I just remember in the DOS days I used a compiler from borland c++ that would generate .exe files.. so it made me think maybe there is some open source compiler that would work with OS/2, from one of those other projects.
Title: Re: Curious what the problem is making new firefox versions.
Post by: Dave Yeo on December 25, 2020, 03:17:08 am
There's a couple of steps in building an executable, including a DLL. First the compiler turns source code into object files. For a large project there are lots of source files, arranged in subdirectories for various functionality. Second the compiler calls a linker to link all those object files into a program or DLL. Sometimes there are other steps involved but we'll ignore them for now.
Another consideration is debug data, which is handy for debugging a program. The debug data can be big as it keeps a bunch of data in the source files and program so that when run in a debugger, you can see the original text code as well as variables etc.
First problem is that sometimes compiling those source files with debug data can use upwards of 2GB's of memory, the workaround there is to compile one at a time, which is wasteful on a multi-core machine and adds time. Firefox here on a 3.1 GHz I5 takes over a hundred minutes to build when compiling up to 5 files at a time, perhaps twice as long or longer when doing one at a time. So by trading off time we can get around this bottleneck.
The next step is feeding all those files into a linker. The linker has to keep track of a lot of stuff as it combines them into one file and can get pretty memory hungry. Firefox's xul.dll is 75 MB's at this point and the QT5 web one much bigger. Even with Mozilla, we had the IBM linker start to fail back at Firefox 4 and had to switch to the OpenWatcom (OW) linker, which we do have the source for and Steven knows how it works and has commit access to put fixes back upstream.
The OW linker also had problems and has been adjusted a few times, it can use what it calls a spill file (think swap) and this is what Roderick is talking about when he talks about modifying wlink the OW linker. So basically we're, or rather some developers, are doing what you suggest. GCC is an open source cross platform current compiler that we have the source for and can adjust and wlink is a very good linker, which once again we have the source for and can adjust. It all takes time from an expert though and gets harder when and if you have to juggle parts into our limited address space.
Then there's the memory that a program uses when running, and a browser can use a lot.
Title: Re: Curious what the problem is making new firefox versions.
Post by: David Kiley on January 29, 2021, 10:41:54 am
It all takes time from an expert though and gets harder when and if you have to juggle parts into our limited address space.
Thanks for the explanation - I had no idea how involved it was :).
And thanks for the work of all the experts making it happen.