Author Topic: LarsenCommander - new test version  (Read 44062 times)

Rich Walsh

  • Sr. Member
  • ****
  • Posts: 398
  • Karma: +30/-0
  • ONU! (OS/2 is NOT Unix!)
    • View Profile
Re: LarsenCommander - new test version
« Reply #30 on: April 22, 2024, 10:29:43 pm »
I would like to have something that saves me some time grabbing by default font size (as XWP shows as default), but I understand that it is not easy to implement for each component described in the Font settings of lcmd.

It shouldn't be particularly hard: there are 5 entries under 'os2.ini->PM_SystemFonts'  [see below] and you map them to your categories. For example, 'PM_SystemFonts->Menus' maps to lcmd's 'AppWin.MenuBar' and its other menu settings.

DefaultFont: System Proportional - if this entry is missing/empty, assume 10 point SysPro

IconText:  WarpSans - this is mislabelled, it should be "Folder text" since it also is used for Details view
WindowText: WarpSans - standard dialog text

Menus: WarpSans Bold - both menubars and popups
WindowTitles:  WarpSans Bold

The key to using defaults while still allowing customization is *never* save the default value. That way you don't get stuck with a value that _used_ to be the default but isn't any longer.
« Last Edit: April 22, 2024, 10:37:00 pm by Rich Walsh »

Rich Walsh

  • Sr. Member
  • ****
  • Posts: 398
  • Karma: +30/-0
  • ONU! (OS/2 is NOT Unix!)
    • View Profile
Re: LarsenCommander - new test version
« Reply #31 on: April 23, 2024, 12:18:14 am »
there are 5 entries [...] and you map them to your categories.

FWIW... you're not limited to just the 3 default fonts. Rather, you can use these entries as guidelines for setting the correct size of your preferred typeface. For example, if lcmd uses 'Helv' somewhere, you can get the entry for the category (say, "menu") but only pay attention to the size (use 'atol(...)). If 'Helv' has that point size, use it. Otherwise, go up or down at point _or_ substitute WarpSans (non-bold).

Do all this during init and save the results so you don't have to duplicate a lot of code. (Note: the default values in 'os2.ini' will never change during a session - they're only updated during shutdown - so there's no need to check for  new values.)

Andi B.

  • Hero Member
  • *****
  • Posts: 911
  • Karma: +16/-2
    • View Profile
Re: LarsenCommander - new test version
« Reply #32 on: April 23, 2024, 04:56:30 pm »
I understand the problem for people with high resolution screen usually found in modern Notebooks. But I think it should be possible with current version to set all controls to the needed bigger fonts. This setting should survive restarts. I'm I right Martin?

If yes I think it's not high priority to invest much time in this and so there's nothing which I can do in near future. F.i. the (rare) crashes and some leak in currently unknown system resources are more important IMO.

I just wait on Martins response and if there is no real show stopper (f.i. some controls can't be changed to usable fonts) I'll create a ticket with all the valuable infos from Rich. Maybe some time someone finds the time to make a more elegant solution for that.

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 5256
  • Karma: +44/-1
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: LarsenCommander - new test version
« Reply #33 on: April 23, 2024, 09:10:17 pm »
Hi Andi B.
I understand the problem for people with high resolution screen usually found in modern Notebooks. But I think it should be possible with current version to set all controls to the needed bigger fonts. This setting should survive restarts. I'm I right Martin?

If yes I think it's not high priority to invest much time in this and so there's nothing which I can do in near future. F.i. the (rare) crashes and some leak in currently unknown system resources are more important IMO.

I just wait on Martins response and if there is no real show stopper (f.i. some controls can't be changed to usable fonts) I'll create a ticket with all the valuable infos from Rich. Maybe some time someone finds the time to make a more elegant solution for that.
Like you said, this is not high priority. For me it is a "nice to have", it is not a bug. If there are no resources to do this, I think it is fine just to document it with a ticket.

Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Dariusz Piatkowski

  • Hero Member
  • *****
  • Posts: 1434
  • Karma: +28/-0
    • View Profile
Re: LarsenCommander - new test version
« Reply #34 on: April 28, 2024, 06:05:07 pm »
Hi Andi,
...
File xfer performance wise, here are the results for movement of a test 800M AVI file:

1) from SAMBA share to ramdisk
OLD:  Peak=58,270K, Avg=43,550K
NEW: Peak=56,220K, Avg=46,516K

...so I'm thinking this is more about being network bound (1G LAN, but of course the Samba share box - NAS is the limitation here)

2) from ramdisk to SSD drive (Samsung 860 Evo, SATA)
OLD:  Peak= 82,788K, Avg=  45,545K
NEW: Peak=310,118K, Avg=220,910K
...

Repeat of the same performance tests for the newest version:

1) from SAMBA share to ramdisk
OLD: Peak=56,220K, Avg=46,516K
NEW: Peak=51,367K, Avg=45,832K

2) from ramdisk to SSD drive (Samsung 860 Evo, SATA)
OLD: Peak=310,118K, Avg=220,910K
NEW: Peak=503,901K, Avg=311,825K

...which naturally made my eyeballs nearly pop out b/c let's be honest, my SSD isn't that fast!!!

So a re-do produced the following NEW:
NEW: Peak=430,674K, Avg=183,007K

...which while still pretty high for a Peak, I suspect this has everything to do with my crazy ass JFS cache, which is 1G in size.

Couple more tests later though the higher Peak appears to be sustained, so perhaps there is a little bit more improvement here Andi!!!?

Thank you again!

Andi B.

  • Hero Member
  • *****
  • Posts: 911
  • Karma: +16/-2
    • View Profile
Re: LarsenCommander - new test version
« Reply #35 on: April 28, 2024, 07:44:28 pm »
Speed tests with SSDs vary on my system. I've tested with -
Samsung SSD 850 PRO 256GB
Samsung SSD 870 EVO 2TB
Samsung SSD 860 QVO 1TB
Crucial CT500MX500SSD1

It's sometimes not clear why copying from one SSD to another is faster with some setting while it's a bit slower with another target or source. I ever thought the maximum throughput you can get with an OS/2 system is copy from CLI cause a GUI program with progress indicator will always lower performance (the progress indicator needs to be update while CLI copy only waits for disk to accept new data, CLI does not even check CTRL-C). To my findings this is still true with disks although now LCMD comes very close to CLI. But in the last months I've even found scenarios where copying with LCMD is much faster than CLI copy (which TTBOMK uses 4k fixed buffer).

You may think copying with LCMD should be faster when not much else is running at the same time. But I've even scenarios where copying is actually a bit faster when LCMD lost the focus and some other heavy task are running. Once I even need a reboot to get the copy speed from/to a specific source/target to the level I'm used to. It was about 20% slower for unknown reason before.

Bottom line is it needs a lot of tests to get reproducible results. In most scenarios you can't get the same full speed with LCMD than with CLI but we are very close to that. Facts based on my tests for single big files are -
- Samba is way slower than ftp (more than 1,5 times slower here), both via netdrive *)
- for SSDs and harddisks LCMD is not exactly as fast als CLI copy but very close to it
- Ramdisk is way slower than SSD or harddisks (JFS is faster than HPFS but still 3 times slower)
- Peak values may be very high cause of SSD (and JFS) cache

*) Samba is way faster than FTP with a lot of small files I think because of it's directory caching which FTP is missing

Andi B.

  • Hero Member
  • *****
  • Posts: 911
  • Karma: +16/-2
    • View Profile
Re: LarsenCommander - new test version
« Reply #36 on: February 23, 2025, 04:22:34 pm »
I've just released v1.09.00. See https://sourceforge.net/projects/lcmd-git/files/

For me this is a major improvement as now the EA logic finally works without memory leaks. At least for my (extensive) testing. Also the focus switching problem improved a lot. Maybe not fully solved but at least the test cases I had. The copy buffer size limit setting now will be honored again. Mind now in MBytes not KBytes. Not sure if translations in all languages are correct.

Unfortunately there is still one problem I couldn't solve. But during extensive testing and debugging I'm pretty sure this is a file system driver (JFS) problem. Copying huge amount of files can lead to file system hangs. Sometimes after 80'000 files, sometimes after 800'000 files. First I thought it's a problem with log file overflow (beyond 2GB). But now as I changed this to LFS support too and numerous test with and without logging file enabled I'm pretty sure it's something very low level in JFS or NVME or ? driver. Given that no one else mentioned this problem until now and given 1.09 is able to copy at least 10 times more files without problems than any version before I think it's time to make it public. Enjoy v1.09.00.

Code: [Select]
Change History:
---------------
20250222 v1.9.0
- Improved Extended Attribute handling again. Now it should handle all sizes of EAs and checks and limits to <64k
- EA handling now really fixes memory leaks
- Reenabled max copy buffer size setting in but now use MB instead of KB because it is in HigherMem arena since
1.8.0 (?) anyway
- Improved focus switching logic. Don not set focus on unrelated windows after copy/del operations.
- Skip sending lot of GUI (PM) messages for updating progress dialog with fast disks while copying and deleting files.
This is to work around PM memory leaks (crashes) when hammered with very much messages in a short period of time.
With fast disks copying or deleting small files takes few ms or even less. The previous logic to update the
progress bar after each file (and during a file when it is bigger) is changed to not send more than about one
message in 125ms. PM seems to be not safely ignore to much unhandled messages and starts to eat up shared memory.
Eventually this crashes the whole system when the process is not closed before. F.i. this skipps more than 69000
messages when copying the ApacheOpenOffice source code tree. The same skipping logic improves speed when deleting lot
of files. F.e. 33s instead 53s for AOO source code tree (about 70k files). More than 76000 messages where skipped with
that new delete logic.
- Change LOG file handling to LargeFileSupport to allow more than 2GB log files. This is needed cause with very heavy
logging previous versions hang the whole file system starting with the log file directory at 2GB log file size
(> 100k files copied with DEBUG)

Mentore

  • Full Member
  • ***
  • Posts: 243
  • Karma: +13/-0
    • View Profile
Re: LarsenCommander - new test version
« Reply #37 on: February 24, 2025, 08:52:54 am »
I've just released v1.09.00. See https://sourceforge.net/projects/lcmd-git/files/


Hi Andi, I'm about to test it  right now. I'm heavily using LC, since I'm so used to the old Norton Commander interface. I also would like to ask you if it's possible to add an "Open With" option on files.
Since I use it really much when I develop or port OS/2 software, something like that would be of great help in order to let me, say, open a source code file with an appropriate editor like VisualSlickEdit, EPM, plain old E, FTE, or maybe a more complex IDE like VisualAge or OpenWatcom.
Was that already on your mind?
In any case, thanks for your good work on this file manager, I feel really at home using it.

Mentore

TeLLie

  • Sr. Member
  • ****
  • Posts: 278
  • Karma: +16/-0
    • View Profile
Re: LarsenCommander - new test version
« Reply #38 on: February 24, 2025, 03:12:42 pm »
Hi Andi,

Thankz for this new version :)

greetz TeLLie


Lars

  • Hero Member
  • *****
  • Posts: 1428
  • Karma: +72/-0
    • View Profile
Re: LarsenCommander - new test version
« Reply #39 on: February 25, 2025, 10:50:33 am »
I've just released v1.09.00. See https://sourceforge.net/projects/lcmd-git/files/

For me this is a major improvement as now the EA logic finally works without memory leaks. At least for my (extensive) testing. Also the focus switching problem improved a lot. Maybe not fully solved but at least the test cases I had. The copy buffer size limit setting now will be honored again. Mind now in MBytes not KBytes. Not sure if translations in all languages are correct.

Unfortunately there is still one problem I couldn't solve. But during extensive testing and debugging I'm pretty sure this is a file system driver (JFS) problem. Copying huge amount of files can lead to file system hangs. Sometimes after 80'000 files, sometimes after 800'000 files. First I thought it's a problem with log file overflow (beyond 2GB). But now as I changed this to LFS support too and numerous test with and without logging file enabled I'm pretty sure it's something very low level in JFS or NVME or ? driver. Given that no one else mentioned this problem until now and given 1.09 is able to copy at least 10 times more files without problems than any version before I think it's time to make it public. Enjoy v1.09.00.

I would suspect this to be a JFS problem: https://github.com/bitwiseworks/libcx/issues/36

Unfortunately, I think "breaking up a large transfer" would mean that the copying application has to do it (unless you are using libcx and the gcc compiler and not calling the OS/2 API directly). If possible and as a test case, you could create a HPFS partition and copy from that and see if that avoids the problem.
« Last Edit: February 25, 2025, 11:23:04 am by Lars »

Andi B.

  • Hero Member
  • *****
  • Posts: 911
  • Karma: +16/-2
    • View Profile
Re: LarsenCommander - new test version
« Reply #40 on: February 25, 2025, 11:43:27 am »
Interesting. But if I understand it correctly it should be fixed since a while. I can reproduce problems here even with ArcaOS5.1EN.

OTOH LarsenCommander doing the copy stuff a bit unusual so this may be another JFS problem others rarely encounter. Lcmd reads the file (in chunks if bigger than internal dynamic buffer) and writes it to target. So far so good. Then it reads in all EAs from source file and then writes the EAs to the target. This EA handling is strange when you see it the first time. Reading every EA name and data with dynamic size.... And I think the problem occurs when writting the EAs.

Unfortunately I didn't find a way on OS/2 to read and write all EAs of a file at once. These can be <64k anyway so no would be no problem on todays systems. Reading all these EA entries in a loop by name and values is a mess.

Lars

  • Hero Member
  • *****
  • Posts: 1428
  • Karma: +72/-0
    • View Profile
Re: LarsenCommander - new test version
« Reply #41 on: February 26, 2025, 12:39:12 am »
Quote
Interesting. But if I understand it correctly it should be fixed since a while. I can reproduce problems here even with ArcaOS5.1EN.
A workaround is in place if you use "read" and "write" Posix functions as these break up large transfers into multiple smaller transfers before they eventually invoke DosRead (DosWrite). The native OS/2 API call "DosRead" continues to suffer from JFS quirks and limitations. It's the IFS driver that is broken, not the DosRead API function.


EAs: DosQueryFileInfo, DosQueryPathInfo, DosSetFileInfo, DosSetPathInfo, "pInfo" pointing to EAOP2 structure:

Getting all EAS at once is done with info level = FIL_QUERYALLEAS = 4.
fpFEA2LIST pointer will point to list of all EAs queried, including their values. You will need to hop from one entry to the next following the oNextEntryOffset field (the individual FEA2s are varying in length). Provide a sufficiently large buffer in fpFEA2List (> 64k, better twice that size) so that it can be filled.

Setting all EAS at once is done with info level = FIL_SETEASFROMLIST = 2.
fpFEA2LIST pointer points to list of all EAS to set, including the values to set. Need to set the oNextEntryOffset to point to the next element (the individual FEA2s are varying in length). Provide a sufficiently large buffer in fpFEA2List (holding all your EAs). You have to ensure that you do not exceed maximum EA overall size.

These info levels are undocumented in the CPS.

You should be able to feed the result of FIL_QUERYALLEAS (read EAs with DosQueryXXXXInfo) into FIL_SETEASFROMLIST (write EAs with DosSetXXXXInfo).

DosEnumAttribute will give you a list of EAs (names) that exist. It's not really needed if you just make the EA buffer large enough (we know the maximum but take into account EA structure and EA name lengths, not only values).
« Last Edit: February 26, 2025, 01:31:04 am by Lars »

Andi B.

  • Hero Member
  • *****
  • Posts: 911
  • Karma: +16/-2
    • View Profile
Re: LarsenCommander - new test version
« Reply #42 on: February 26, 2025, 08:37:02 am »
Quote
These info levels are undocumented in the CPS.
Wow. Some of your explanations sounds familiar. After I invested countless hours on reading original Lcmd code, edm2 and all the help files of os2tk45. But where did you get the undocumented stuff from? Answers from some experts in some goups which you noted somewhere? Or is there a place where someone document such undocumented stuff? Stevens site?

Btw: these idiots which wrote the control programming appendum for DosOpenL f.i. should be sent to jail. Again I stumbled over this wrong description and examples (variables versus pointers). It's a mess when the official documentation is so wrong in so many places :-(.

Btw2: It's a somehow educated guess that the file system is faulty here. With different versions of EA handling in Lcmd I very often got 'Trap 000e in JFS'. The last implementation which I think perfectly correct, let the IFS silently stop working only sometimes. Unfortunately I need to copy a few hundred thousand files with a lot of GBs. So still have no simple scenario to reproduce.

Btw3: For my problem I think it's not the DosRead buffer size as this is already limited (<1 - ~16MB) here. AFAIK the problem Silvan reported happens with about 400MB and above. OTOH David explained in the ticket it's not the buffer size which triggers this one known problem, but 2 DosOpenL calls where the first one does nothing. Do you think Davids explanation in the ticket is wrong? Or is he talking about another problem? Is it worth to test with read buffers max. 1MB f.i.?

Lars

  • Hero Member
  • *****
  • Posts: 1428
  • Karma: +72/-0
    • View Profile
Re: LarsenCommander - new test version
« Reply #43 on: February 26, 2025, 09:57:29 am »
Hi Andi,

Quote
These info levels are undocumented in the CPS.
Wow. Some of your explanations sounds familiar. After I invested countless hours on reading original Lcmd code, edm2 and all the help files of os2tk45. But where did you get the undocumented stuff from? Answers from some experts in some goups which you noted somewhere? Or is there a place where someone document such undocumented stuff? Stevens site?

...

Btw3: For my problem I think it's not the DosRead buffer size as this is already limited (<1 - ~16MB) here. AFAIK the problem Silvan reported happens with about 400MB and above. OTOH David explained in the ticket it's not the buffer size which triggers this one known problem, but 2 DosOpenL calls where the first one does nothing. Do you think Davids explanation in the ticket is wrong? Or is he talking about another problem? Is it worth to test with read buffers max. 1MB f.i.?

For EAs: I looked into the OpenJFS Source code at netlabs.org and what it does. Needless to say that it does stuff that is not documented but I took it by the word (and hope HPFS does the same):

https://trac.netlabs.org/OpenJfs

and I looked at a very old OS/2 Version 1.2 IFS Specification written by Microsoft in 1989 that I found in the internet, it appears as part of European patent EP0415346A2:

https://patentimages.storage.googleapis.com/23/69/45/f6a4cb501938a9/EP0415346A2.pdf

it was scanned by someone from a pile of paper (and that I cleaned from junk and reprocessed as a new PDF to make it more accessible). Don't forget to scroll through it all the way !
I have no clue of why some of the info in the old IFS specification got lost along the way but on the other hand it covers stuff that is no longer relevant/workable, you need to be careful ...

In the old IFS spec, you can read the following for DosQFileInfo / DosQPathInfo:

Quote
*** THE FOLLOWING UNPUBLISHED FUNCTION WILL GO AWAY IN FUTURE *****
*** RELEASES AND SHOULD BE UNPUBLISHED ***
Level 0x0004 file information returns all EA info for the file. On input, FileInfoBuf is an EAOP structure above. fpGEAList contents are ignored. fpFEAList points to a data area where the relevant FEA list will be returned. The length field of this FEA list is valid, giving the size of the FEA list buffer. offError is ignored.
...
Guess what, that info level never went away (translate "DosQueryFileInfo" for "DosQFileInfo"), likely because some app used it and then MS needed to continue supporting it :-)

Likewise for DosSetFileInfo / DosSetPathInfo:

Quote
Level 0x0002 file information sets a series of EA name/value pairs. On input, PathInfoBuf is an EAOP structure above. fpGEAList is ignored. fpFEAList ponts to a data area where the relevant FEA list is to be found. offError is ignored.

I guess that by now, you get the idea :-)


For AN ticket: I no longer have access to the AN ticket system and therefore cannot see what they found out.



« Last Edit: February 26, 2025, 05:03:10 pm by Lars »

Andi B.

  • Hero Member
  • *****
  • Posts: 911
  • Karma: +16/-2
    • View Profile
Re: LarsenCommander - new test version
« Reply #44 on: February 26, 2025, 12:21:27 pm »
Thank you very much. Enough stuff to test and try out. Have to bookmark this. OTOH I thought this lcmd version would be really the last one.....

Ouch, I don't have access to the source code at netlabs :-(.

« Last Edit: February 26, 2025, 12:24:18 pm by Andi B. »