OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Applications => Topic started by: Andi B. on May 19, 2023, 12:07:08 pm

Title: LarsenCommander - new test version
Post by: Andi B. on May 19, 2023, 12:07:08 pm
Since a while I'm working on some problems in LarsenCommander again. There is still one problem I want to resolve before I make an official release. But as I'm preparing for holiday vacation I think it may be good to spread the current version to the public.

Main difference to older releases is - I reenabled big buffers for file copy operations. The advantage is much faster copy speed especially with big files. Copy speed is still not as fast as the copy command from a 4os2 command prompt but the difference is not as dramatic than before. I had to rework and extend the original dynamic buffer adaption code for this. It seems current copy speed is the maximum you can get when copying with an PM application.

The max. copy buffer size in old releases was determined by a simple algorithm and based on the setting in Options - Various - File Copy/move tab. I had to limit the buffer in some 1.7. release to 60k cause there was a subtle bug in ndfs DAV plugin which leads to some zero bytes in large files copied with the help of big buffers. The side effect was a limited copy speed at about 30MBytes/s I just discovered a few months ago. There's an old ticket about the ndpdav bug which is resolved in newer releases I think. So I think it's save to reenable big buffers again. For convenience I include my fixed ndpdav.dll in this lcmd*zip. But you probably don't need it. Current ndfs 3.1.6 also has changed in a way that such subtle plugin bugs shouldn't show up anymore. But be warned if you use old ndfs/ndpdav versions.

Lcmd also uses high memory for the copy buffer now. So if you watch your memory consumption (f.i. with above512.cmd or theseus) you may find a bit less memory usage from the scarce lower shared mem. Unfortunately this does not fix another long standing problem - copying big trees of directories sometimes leads to eat up a lot (sometimes all) lower shared memory. This may eventually bring your system down until you kill lcmd. Finally I can reproduce this problem but unfortunately I don't have a fix for this right now. If you ever run into this bug in the past, sorry there's no fix in this release for that.

In this archive I've also included the current available translations. When you want to use lcmd with another language or make your own translation look into the locale directory.

I've also checked in all current sources at the official home of LarsenCommander - https://sourceforge.net/projects/lcmd/

Have fun :-)
Title: Re: LarsenCommander - new test version
Post by: Martin Iturbide on May 19, 2023, 05:16:23 pm
Hi Andi B.

Thanks for taking the time to update Larsen Commander.

I was trying the new version and I have a little question or issue. When I have a file on the right side, and I want to copy it to the left side (Different folder). I select the file and click "copy", a popup shows telling me that it will copy the file from here to there, I click ok, the file get copied but the desktop windows loose focus, the focus moves around and a window from the background gets to the top. (Maybe the pictures explain my problem if my wording is not clear enough)

It also happens when I delete a file with the right click menu too.  Is that only happening to me? 

Regards
Title: Re: LarsenCommander - new test version
Post by: Andi B. on May 19, 2023, 05:55:41 pm
Maybe you're bitten by the same bug I mentioned in the ToDo.txt file more than 12 years ago ;-) (20100810 After deleting files focus changes to another not lcmd window).

I remember I added some debug messages back then when I tried to track this down. But couldn't solve this. But as this is a nuisance for me too chances are high that I finally will fix it.

 
Title: Re: LarsenCommander - new test version
Post by: Andy Willis on May 20, 2023, 07:28:48 pm
Thank you Andi.
I updated my local svn and built it.  Thought I had a problem with the executable at first, I tried to test the built lcmd.exe in the directory it was built in and it threw errors but I figured out it was reading the lcmd.rc for building it and the executable needs its own lcmd.rc file.  Once I worked that out, it seems to be working well.
One thing that I would like I would like to see (maybe I can work it out) is to have lcmd.exe look for gl16os2.dll in its own directory first before looking at libpath, this would allow it to run without gl16os2.dll having to be in libpath.
Title: Re: LarsenCommander - new test version
Post by: Andi B. on May 26, 2023, 05:48:50 pm
Quote
this would allow it to run without gl16os2.dll having to be in libpath.
I don't think I've it in the libpath. But I will check next week.
Title: Re: LarsenCommander - new test version
Post by: Dariusz Piatkowski on May 26, 2023, 06:05:56 pm
Hello Andi!

As the others have stated: big Thank You!!!

...Main difference to older releases is - I reenabled big buffers for file copy operations...

The max. copy buffer size in old releases was determined by a simple algorithm and based on the setting in Options - Various - File Copy/move tab. I had to limit the buffer in some 1.7. release to 60k cause there was a subtle bug in ndfs DAV plugin which leads to some zero bytes in large files copied with the help of big buffers....

So I'm curious about this: does the current Max Buffer setting still function, or is that now ignored by your enhanced logic?

I had previously set this to 10K, now set to 64K, tested across the local as well as LAN devices (Win boxes as well as a NAS), all still good.

OK, so I do have a 'feature request': WHEN looking at the contents of a ZIP file do you think you might be able to enable object recognition and the use of system-wide settings instead of the default editor that's defined in LCMD settings?

I'm literally thinking:
1) TEXT is fine
2) JPG, GIF, etc...do not work
3) MP4, AVI...do not work

...so am I perhaps missing something here, or is LCMD really limitted to just being able to treat everything inside a ZIP  file as a TEXT file?

Thanks!

EDIT
====

I clarified that we are talking about ZIP file contents here...my original post did NOT spell this out at all!
Title: Re: LarsenCommander - new test version
Post by: Andi B. on June 08, 2023, 11:19:04 am
Thank you Andi.
I updated my local svn and built it.  Thought I had a problem with the executable at first, I tried to test the built lcmd.exe in the directory it was built in and it threw errors but I figured out it was reading the lcmd.rc for building it and the executable needs its own lcmd.rc file.  Once I worked that out, it seems to be working well.
Yes I was confused by this more than once too. There's included the lcmd_text.cmd (and te.cmd) which calls
Code: [Select]
start lcmd.exe -rcFile ..\..\lcmd.rc -rcSearchPath ..\..\src\glib;..\.. -iniFile lcmd.ini -logFilter test -logFile lcmd.log %1 %2 %3 ...
Quote
One thing that I would like I would like to see (maybe I can work it out) is to have lcmd.exe look for gl16os2.dll in its own directory first before looking at libpath, this would allow it to run without gl16os2.dll having to be in libpath.
I've ';.;' in libpath after the %unixroot% directories.
Code: [Select]
LIBPATH=p:\usr\local\lib;p:\usr\lib;p:\dll;.;P:\.....So that's no problem for me. Isn't that the usual setup? I mean having ;.; in the libpath (best at the end or after unixroot dirs. Isn't it the standard behavior for OS/2 programs to search their dlls in libpath? Or start it by an object set to run in the programs directory?

Are you asking for some 'special' code in LCMD which loads the dll from the programs directory regardless of libpath setting? Are there some example programs for such?
Title: Re: LarsenCommander - new test version
Post by: Andi B. on June 08, 2023, 12:06:32 pm
....
So I'm curious about this: does the current Max Buffer setting still function, or is that now ignored by your enhanced logic?
Currently it's only honored when you set it bigger than 16MByte. And only for the first file block which is copied. If the file is bigger than 16MB (or your setting) then the buffer may be increased when copying one block takes less than 40ms. And the buffer size gets decreased when copying a block takes more than 350ms. Actually the 'ideal' copy buffer size calculation is a bit more complex but the idea is to change the buffer size dynamically so that copying one block takes between about 40 - 350ms. This all has a big impact on real big files (lot of 100MBytes) and fast devices (f.i. SSDs or fast HDDs).

Originally the 'ideal' buffer size was calculated for a block transfer time of 4s (if I understand the code correct) but limited to your setting in the options and a rather small initial setting. I had to change that cause some overflow in the calculation leads to wrong buffer size and very slow transfer rates in some cases. Moreover I think most users didn't even find the setting in the options (like myself) and so the got only low transfer rates even with very fast devices.

With my current algorithm I have about 85MBytes/s when writing to HDDs and >140MBytes/s on some SSD. This is close enough to the limit my system is able to handle and I think most I can get out of a PM copy program with progress indication.

I mainly tried to optimize copy speed with big files and reals disks within the system. I've not seen a negative impact in copy speed for small files. Limited testing was done on different LAN or USB devices and different file systems (JFS, HPFS, FAT, FAT32) too without negative side effect. Unfortunately I've currently no chance to test fast USB3 devices. They do not work well on my system in general. So test result with different settings are welcome. Mind you always can compare speed with copy done from cli. Usually copy from cli is still faster. But not that much anymore than before.

Quote
...
OK, so I do have a 'feature request': WHEN looking at the contents of a ZIP file do you think you might be able to enable object recognition and the use of system-wide settings instead of the default editor that's defined in LCMD settings?
...
I consider the vfs implementation (zip file handling) in LCMD as a proof of concept. Only basic functionality is implemented AFAIK. As you already have observed functionality is very limited. I never used this feature by myself. I find even file managers which implement such feature much better (TotalCommander on Win) as rather useless and I don't use it. Cause even there it never works nearly as good as a real Archive program manager (WinRar, 7-zip, Win folder zip integration, ...). I don't see any real chance for me to implement zip/rar/arj/tar/iso/... file handling in LCMD to an extend I by myself would call it nearly useful. Not even if I got fired by my company and could work full time on this ;-). So until no one else could invest a lot of time in this part of LCMD I would suggest setting 'System launch Zip's by default'.
Title: Re: LarsenCommander - new test version
Post by: Andy Willis on June 09, 2023, 04:13:07 am
Thank you Andi.
I updated my local svn and built it.  Thought I had a problem with the executable at first, I tried to test the built lcmd.exe in the directory it was built in and it threw errors but I figured out it was reading the lcmd.rc for building it and the executable needs its own lcmd.rc file.  Once I worked that out, it seems to be working well.
Yes I was confused by this more than once too. There's included the lcmd_text.cmd (and te.cmd) which calls
Code: [Select]
start lcmd.exe -rcFile ..\..\lcmd.rc -rcSearchPath ..\..\src\glib;..\.. -iniFile lcmd.ini -logFilter test -logFile lcmd.log %1 %2 %3 ...
Quote
One thing that I would like I would like to see (maybe I can work it out) is to have lcmd.exe look for gl16os2.dll in its own directory first before looking at libpath, this would allow it to run without gl16os2.dll having to be in libpath.
I've ';.;' in libpath after the %unixroot% directories.
Code: [Select]
LIBPATH=p:\usr\local\lib;p:\usr\lib;p:\dll;.;P:\.....So that's no problem for me. Isn't that the usual setup? I mean having ;.; in the libpath (best at the end or after unixroot dirs. Isn't it the standard behavior for OS/2 programs to search their dlls in libpath? Or start it by an object set to run in the programs directory?

Are you asking for some 'special' code in LCMD which loads the dll from the programs directory regardless of libpath setting? Are there some example programs for such?
I have lcmd in path but am trying to avoid having to put the dll in the libpath.  There is a way when calling the DLL to search the path the the executable is in first, ;.; only helps if you are in the directory that the DLL is in when you launch the executable.  I should be able to find the code in Mozilla.
Title: Re: LarsenCommander - new test version
Post by: Andi B. on June 09, 2023, 10:00:21 am

I have lcmd in path but am trying to avoid having to put the dll in the libpath.  There is a way when calling the DLL to search the path the the executable is in first, ;.; only helps if you are in the directory that the DLL is in when you launch the executable.  I should be able to find the code in Mozilla.
[/quote]
Ok now I understand. Different people, different habits. I don't have the LCMD directory in PATH nor in LIBPATH. I've three different objects in xcenter which starts LCMD with different settings and at least one object on the desktop. Looking at it I now see one of them points to te.cmd. Others have '*' as 'Path and file name' and something like '/c set beginlibpath="P:\Util\Larsen Commander_OwnTest" && "P:\Util\Larsen Commander_OwnTest\lcmd.exe" -rcSearchPath P:\Util\Larsen Commander_OwnTest -rightDir E:\_work -leftDir E:\Dokumente -iniFile E:\home\lcm_own.ini' as Parameters. Now I see why I don't need it to bee in any (LIB)PATH :-)

If I have a need to start a program from cli then I usually have a *.cmd file in my default \batch file directory (which is in PATH). With this habit (objects in xcenters and .cmd files) for a lot of programs I keep my PATH and LIBPATH short. As said different users different needs.

Anyway if you find some solution for your problem maybe we can implement it.
Title: Re: LarsenCommander - new test version
Post by: Rich Walsh on June 10, 2023, 05:45:41 am
One thing that I would like I would like to see (maybe I can work it out) is to have lcmd.exe look for gl16os2.dll in its own directory first before looking at libpath, this would allow it to run without gl16os2.dll having to be in libpath.
Are you asking for some 'special' code in LCMD which loads the dll from the programs directory regardless of libpath setting? Are there some example programs for such?

The solution to this particular problem is to get rid of the DLL -- there's no reaon for it to exist. The original author claimed that VACPP 4.0 couldn't handle a 16-bit call so he created a separately-compiled file. But rather than simply linking it into the exe with all the other .obj files, he created a dll. Why?

VAC 3.65 has no problem with 16-bit calls, and AIUI, 3.65 is just 4.0 without the class libraries. I'd expect you could compile this file the same as all the others, then add it to the ilink cmd line. Even if you do need to use a different compiler, it should still link OK.
Title: Re: LarsenCommander - new test version
Post by: Lars on June 11, 2023, 03:11:24 pm
I think, this misunderstanding stems from the fact that every IOPL example (which means 16-bit code use by the nature of IOPL support in OS/2) seems to imply that you need to put the IOPL code into a separate DLL. Well, you do not have to do that, just needs some additions to the def file and you can happily link 16-bit IOPL code to the rest of your 32-bit app.
Title: Re: LarsenCommander - new test version
Post by: Andi B. on June 29, 2023, 10:03:18 am
Thanks Rich and Lars. I'm thinking about your inputs and hope to find the time to come back to this.

Rich, to my understanding VAC4 is a complete different thing. It uses vacbld which creates a code store and compiles all sources at once (?) instead classic icc for every file and makefiles. But in the package of VAC4 there's also included v3.65 with a lot of information when you can or should switch from older versions to 4.0 or to 3.65. I read them long ago and probably forgot most of them. Anyway I'm thinking about switching to gcc since a while. But it's not that easy and so not top priority.

Currently I try to track down a memory leak in LCMD in some situations. Doing this I stumbled over a shortcoming in pmprintf which eats up all lower shared memory when loaded very heavy. So I try to improve pmprintf first (make use of high mem). While doing this I was bitten by a shortcoming of our os2ahci driver. Reproducing this problem, creating a ticket and help to track it down took a lot of time. So it's all about having to less time for all these interesting stuff to get finally solved. So better not giving any time frame....
Title: Re: LarsenCommander - new test version
Post by: Andi B. on November 30, 2023, 07:13:48 pm
I've upload a new version here - https://sourceforge.net/projects/lcmd-git/files/

Mind this is a new project page at sourceforge as I'm using git instead svn now.

This is a lxlite stripped down debug version with the debug information in the *.dbg file. In case of an exception exceptq should be able to create a nice report. At least when the lcmd build in exception handlers will not catch it before (ToDo for future releases). Although debug symbols are compiled in, pmprintf.dll should not be needed for this build.

Changes from last release are -
- Fine tuning for dynamic copy buffer size adoption
- Reduce logging while copying huge amount of files (reduces memory needs and log file size)
- Disable playing priority games in hope of better overall system response
- Adding ExceptQ handler (not working yet as own exception is still active)
- Fixed some cppcheck warnings (thanks Andy Willis for that)
- Add some comments for using highmem in some places (currently not used)
- Add writing to PmPrintf queue in parallel to log file when #ifdef __DEBUG_TRACE__
- Use HighMem for copy buffer

Especially the tuning of the copy buffer algorithm makes a big difference in copy speed in so some cases. At least with my rather fast HDDs and SSDs and big files. But also with Samba shares or other source/targets differences can be observed. All my measurements gave good results. I think this is near to the maximum copy speed we can reach with an PM OS/2 application with progress bar. If you think copy speed now is slower than with any previous version please share your experience here with clear instructions to reproduce. I really tested a lot scenarios with different targets and think the current implementation is better then everything before. Mind 'copy' from command line is most of the time still a bit faster as it does not have the overhead of a graphical progress bar and no checks for interruption during copying a file.

Get ride of gl16os2.dll is still on the ToDo list though.
Title: Re: LarsenCommander - new test version
Post by: Dariusz Piatkowski on December 02, 2023, 08:12:35 pm
Hi Andi,

I've upload a new version here - https://sourceforge.net/projects/lcmd-git/files/
...
Changes from last release are -
- Fine tuning for dynamic copy buffer size adoption
- Reduce logging while copying huge amount of files (reduces memory needs and log file size)
- Disable playing priority games in hope of better overall system response
- Adding ExceptQ handler (not working yet as own exception is still active)
- Fixed some cppcheck warnings (thanks Andy Willis for that)
- Add some comments for using highmem in some places (currently not used)
- Add writing to PmPrintf queue in parallel to log file when #ifdef __DEBUG_TRACE__
- Use HighMem for copy buffer

Especially the tuning of the copy buffer algorithm makes a big difference in copy speed in so some cases. At least with my rather fast HDDs and SSDs and big files. But also with Samba shares or other source/targets differences can be observed. All my measurements gave good results...

Application functionality seems to be spot-on.

File xfer performance wise, here are the results for movement of a test 800M AVI file:

1) from SAMBA share to ramdisk
OLD:  Peak=58,270K, Avg=43,550K
NEW: Peak=56,220K, Avg=46,516K

...so I'm thinking this is more about being network bound (1G LAN, but of course the Samba share box - NAS is the limitation here)

2) from ramdisk to SSD drive (Samsung 860 Evo, SATA)
OLD:  Peak= 82,788K, Avg=  45,545K
NEW: Peak=310,118K, Avg=220,910K

...here I'm thinking there is an OBVIOUS and huge difference! In fact I was floored by how much of a change this was, pretty much maxing out the throughput I would normally see on my DISKIO tests, so I'd say this is an extremely good result Andi!!! Cheers to you sir.

Now, a couple of further remarks as I'm wondering if you would consider looking into these further:

1) NOT an obivous bug, but maybe one???
Opening up a MP4 file causes my MPLAYER app to be started, but because I actually have a CMD that drives it (which accepts a parameter - [Input Scaling Factor] "%*" - that pops up as that INPUT box which allows me to input a scaling factor) the video starts playing, but I notice that the MPLAYER app actually was started twice.

This is why I'm thinking it maybe a bug...and perhaps only dealing with instantiating CMD started apps, because doing the very same thing for PDF files for example opens up a single instance of Lucide.

2) file type specific icons
Let me use the example of JPGs. If these have the icon stored as EA, LCMD pulls these in and shows them. However my NAS does not support EAs, and therefore all of the pics simply show a document icon as opposed to the DEFAULT PMVIEW icon.

This behavious applies to numerous other filetypes as well.

Would it therefore be possible to modify LCMD to use default APP-specific ICONS for particular filetype when the EAs do not have the icon information available?

Once again, thank you for maintaining and enhancing LCMD!!!
Title: Re: LarsenCommander - new test version
Post by: Andi B. on December 03, 2023, 11:13:01 am
Just a quick not about -
Quote
1) from SAMBA share to ramdisk
OLD:  Peak=58,270K, Avg=43,550K
NEW: Peak=56,220K, Avg=46,516K

...so I'm thinking this is more about being network bound (1G LAN, but of course the Samba share box - NAS is the limitation here)

2) from ramdisk to SSD drive (Samsung 860 Evo, SATA)
OLD:  Peak= 82,788K, Avg=  45,545K
NEW: Peak=310,118K, Avg=220,910K

On Samba shares I can not get more then about 36MBytes/s. No matter what the other system is running. I think this is limited by our netdrive/samba client setup. It's not the GBit network. You can check with ftp to the same system and will get something more close to 1GBit (80-90MByte/s IIRC).

Ramdisk - writting to our RAMDISK is not more than about 55MBytes/s here. It would be faster when using strategy 1 but this would limit to <2GB and does have other big problems which is the reason I stick without /1.

I wonder about your really fast SATA performance. I've tested here with Samsung 850 Pro, Samsung EVO 870, Samsung 860 QVD and various HDDs including my WD black. It seems my 9 year old 4 core system can not handle more than about 160MBytes/s. So I currently I can't test with systems as fast as yours.
Title: Re: LarsenCommander - new test version
Post by: Dariusz Piatkowski on December 03, 2023, 06:53:15 pm
Andi,

...
On Samba shares I can not get more then about 36MBytes/s. No matter what the other system is running. I think this is limited by our netdrive/samba client setup. It's not the GBit network. You can check with ftp to the same system and will get something more close to 1GBit (80-90MByte/s IIRC).

Totally agree, here is what Kai Uwe Rommel's excellent NETIO utility reports ('orclsrvr' is my Win7Pro box on the LAN):

Code: [Select]
[G:\]netio -t orclsrvr

NETIO - Network Throughput Benchmark, Version 1.30
(C) 1997-2008 Kai Uwe Rommel

TCP connection established.
Packet size  1k bytes:  108.65 MByte/s Tx,  82.77 MByte/s Rx.
Packet size  2k bytes:  109.66 MByte/s Tx,  108.81 MByte/s Rx.
Packet size  4k bytes:  108.43 MByte/s Tx,  111.03 MByte/s Rx.
Packet size  8k bytes:  106.60 MByte/s Tx,  105.70 MByte/s Rx.
Packet size 16k bytes:  104.48 MByte/s Tx,  9.06 MByte/s Rx.
Packet size 32k bytes:  110.89 MByte/s Tx,  111.15 MByte/s Rx.

So the pipe is big enough, althought the severe drop with the 16k packet is always there, but otherwise it's pretty much using the full Gig.

Just to stay consistent in my testing (given that the network throughput is being tested against a different box - not the NAS one) I re-ran the same test but just with the NEW LCMD version:

1) from SAMBA WIN7 share to ramdisk
NEW: Peak=74,012K, Avg=50,996K

The NAS box is just a little slower, although I do NOT see this when moving stuff between the Win7 and NAS boxes, so that tells me it is indeed our Samba client.

...Ramdisk - writting to our RAMDISK is not more than about 55MBytes/s here. It would be faster when using strategy 1 but this would limit to <2GB and does have other big problems which is the reason I stick without /1.

I wonder about your really fast SATA performance. I've tested here with Samsung 850 Pro, Samsung EVO 870, Samsung 860 QVD and various HDDs including my WD black. It seems my 9 year old 4 core system can not handle more than about 160MBytes/s. So I currently I can't test with systems as fast as yours.

For my RAMDISK I just have the following in my CONFIG.SYS:

'BASEDEV=HD4DISK.ADD /V'

...and my QSSETUP.CMD shows:

'ramdisk Y: jfs'

It's been a while since I was testing the performance of that configuration...probably not since I moved from HPFS386 to JFS actually.

Here are the DISKIO metrics for my SSDs:

1) Evo 850
Quote
[G:\]diskio -hd 2
DISKIO - Fixed Disk Benchmark, Version 1.20

Dhrystone 2.1 C benchmark routines (C) 1988 Reinhold P. Weicker
Dhrystone benchmark for this CPU: 6089057 runs/sec

Hard disk 2: 255 sides, 30401 cylinders, 63 sectors per track = 238472 MB
Drive cache/bus transfer rate: 129526 k/sec
Data transfer rate on cylinder 0   : 349038 k/sec
Data transfer rate on cylinder 30399: 318076 k/sec
CPU usage by full speed disk transfers: 6%
Average latency time: 0.1 ms
Average data access time: Disk read error.

Multithreaded disk I/O (4 threads):
 124476 k/sec, 4% CPU usage

and here is the Evo 860
Quote
[G:\]diskio -hd 4
DISKIO - Fixed Disk Benchmark, Version 1.20

Dhrystone 2.1 C benchmark routines (C) 1988 Reinhold P. Weicker
Dhrystone benchmark for this CPU: 6047660 runs/sec

Hard disk 4: 255 sides, 15960 cylinders, 240 sectors per track = 476929 MB
Drive cache/bus transfer rate: 151264 k/sec
Data transfer rate on cylinder 0   : 373430 k/sec
Data transfer rate on cylinder 15958: 384912 k/sec
CPU usage by full speed disk transfers: 21%
Average latency time: 0.0 ms
Average data access time: 0.3 ms

Multithreaded disk I/O (4 threads):
 216995 k/sec, 29% CPU usage

My box is pretty ancient by today's standards as well (MSI 890FXA-GD70), but it has SATA3 SB850 controller, which did make a difference when I moved to that motherboard from my old SATA2 controller one.

...but anyways, that's just a tanget to this core LCMD changes feedback.
Title: Re: LarsenCommander - new test version
Post by: Andy Willis on December 05, 2023, 06:45:03 am
Thank you for your work.
I pulled the source from git and find there were build changes:
call envtk45.cmd -noansi
call envicc40.cmd $ -noansi
These cmd files are not in the repository and I do not have them in my build environment.  Are these files you built or should I have them in my environment?

Title: Re: LarsenCommander - new test version
Post by: Andi B. on December 05, 2023, 09:20:58 am
Thank you for your work.
I pulled the source from git and find there were build changes:
call envtk45.cmd -noansi
call envicc40.cmd $ -noansi
These cmd files are not in the repository and I do not have them in my build environment.  Are these files you built or should I have them in my environment?
Thanks for testing the new repository. Obviously I've checked in some local changes I made long ago which I have never updated in the svn.
envtk45.cmd and envicc40.cmd are my local files similar to the setenv* commands for toolkit 45 and vacpp40. But instead changing to fixed inc/lib/paths I check before if the needed files are accessible and only add the new paths when they are not already set. So this is only to ensure that the proper toolkit and compiler environment is set.

I wonder if I should add these files to the repository cause they are kind of quick and dirty REXX scripts. And they include my fixed drive/path settings. Maybe I should rem them out in the repo. If you wonder about the -noansi - this is for running them within Slickedit to suppress the colored output which is funny for running from 4os2. I think in the Linux world the -color option would be the more common term for such behavior. You may found out I usually compile within Visual Slickedit calling vs_make.cmd by simply pressing <F5> (Project properties - Build options....).
Title: Re: LarsenCommander - new test version
Post by: Alfredo Fernández Díaz on December 05, 2023, 03:32:36 pm
I'm glad to to see LC actively developed once again. Thank you, Andi - will be giving it a spin : )
Quick question - is NLS functional?
Title: Re: LarsenCommander - new test version
Post by: Andi B. on December 08, 2023, 10:58:16 am
I'm glad to to see LC actively developed once again. Thank you, Andi - will be giving it a spin : )
Quick question - is NLS functional?
It should be all included in the package. I remember I've worked on adding all necessary components for NLS to the distribution script long time ago. But I didn't test it. So feedback is appreciated.

Remember at https://sourceforge.net/p/lcmd-git/tickets/ there's a ticket system. Although I try to read all infos here, chances that something get remembered in future is much higher when there's a ticket ;-). No clue if you need an account for that. For me sourceforge works pretty good with our ancient SM (and so FF I think). In contrary to other sites username/password authentication still works.
Title: Re: LarsenCommander - new test version
Post by: Alfredo Fernández Díaz on December 08, 2023, 08:23:49 pm
Hi again, Andi, thank you...
It should be all included in the package.

I asked whether it is functional because I saw a bunch of old RC translated files under the 'locale' subdir with no obvious way to switch languages from within the application (and still can't find it).

Quote
I remember I've worked on adding all necessary components for NLS to the distribution script long time ago. But I didn't test it. So feedback is appreciated.

OK, fiddling about a bit, under locale there is an old email called 'reply.txt' that you can get rid of. Its slightly more updated cousin 'readme_translation.txt' (which you could refactor) reads "start Larsen Commander version 1.5.1 or 1.5.2 with this command: LCMD.EXE -rcFileLocale lcmd_it.rc" . I cloned the English file onto a Spanish one in the LC folder and changed a few strings -- LC picks it up with the proper parameters in place in the program object. Functional, but a bit cumbersome to keep switching languages while translation is in progress.

Quote
Remember at https://sourceforge.net/p/lcmd-git/tickets/ there's a ticket system. Although I try to read all infos here, chances that something get remembered in future is much higher when there's a ticket ;-). No clue if you need an account for that. For me sourceforge works pretty good with our ancient SM (and so FF I think). In contrary to other sites username/password authentication still works.

OK, but let me see first if I can figure out the locale format ("text" and "hint" are obvious, but what are "alt1", 2, and 3?), how everything is put together (menus seem to be scattered), and whether the autogenerated mnemonics and such do not make translation much of a PITA.
Title: Re: LarsenCommander - new test version
Post by: Andi B. on December 14, 2023, 10:36:50 pm
The solution to this particular problem is to get rid of the DLL -- there's no reaon for it to exist. The original author claimed that VACPP 4.0 couldn't handle a 16-bit call so he created a separately-compiled file. But rather than simply linking it into the exe with all the other .obj files, he created a dll. Why?
...

It seems it was that easy - add '      source "gl16os2.obj"' to lcmd.icc and don't delete gl16os2.obj during the build. As far as I can see the only 16 bit function which is used by lcmd is Dos16MemAvail. But no clue when it is called. Have to test a while to be sure there's no sideeffect.
Title: Re: LarsenCommander - new test version
Post by: Andi B. on April 19, 2024, 07:41:52 pm
I've just uploaded v1.08.02 at https://sourceforge.net/projects/lcmd-git/files/

From the Readme -
Code: [Select]
Change History:
---------------
20240416 v1.8.2
- Improve EA handling routines to reduce memory leaks and silent crashes
- Fix TRAP 000e in JFS while copying >69000 files. Hint, please use latest ArcaOS (kernel, JFS, ACPI 3.23.17, ...)
- Removed gl16os2.dll dependency (link .obj into .exe)
- Handel out of memory conditions more gracefully
- Added abtrace for logging via pmprintf.dll (if available on target system)
- Improved build cmd files to break on error (ERRORLEVEL handling)
- Tuneed lxlite options to not remove bldlevel info
- Change to David G. icon files (no clue where they are used)
- Again tuning of dynamic copy buffer size adoption (even more speed on my system :-) )
- Change icons to the ones supplied by David G. Thanks for that although it's not clear to me where they are used.
(- Disable integrated exception handlers in favor of exceptq) reverted back
- Disabled ExceptQ handler as I can't get it to work in conjunction with lcmds own handlers

Bug reports please here - https://sourceforge.net/p/lcmd-git/tickets/

Of course I'll keep reading here as well from time to time. Have fun.

Btw. I've a really hard time to track down and reduce this terrible JFS traps when copying a lot of files. It seems to me that JFS still does not touch some memory pages when feed with >4k buffers some times which I guess leads to these 000e traps. Combined with the memory leaks which LCMD suffered when lot of files where copied I had uncountable crashes with long chkdsks afterwards. Now the I reduced memory need for extended attribute handling I can't see these memory leaks anymore with above512. But on some scenarios LCMD still seems to eat up some system resources which report out of memory. Closing LCMD will recover from such situations now.

If you encounter hangs of LCMD you may look at the lcmd.log file in the directory of lcmd.exe. F.i. tail -n 50 lcmd.log will show you the last messages which usually shows some exception infos. For this you've to enable 'Debug' or 'Test' in Help - Debug - Options. Be aware running LCMD with 'Debug' for a long time or while copying huge amount of files let lcmd.log become very big in minutes.

Btw. if you've pmprintf.dll on your system PmPrintf.exe will show you these messages from LCMD with 'Debug' or 'Test' enabled too.
Title: Re: LarsenCommander - new test version
Post by: Andy Willis on April 20, 2024, 02:36:15 am
I have just downloaded it so I haven't had a chance to use it to say anything about it other than to thank you for your work on it. 
Title: Re: LarsenCommander - new test version
Post by: roberto on April 20, 2024, 08:54:18 pm
Hello
Thank you very much for your effort.
 I have a little problem that is very annoying to me.
  Let me explain:
When I have the lcmd open and on top of that I have the editor e.exe open
 I MOVE my mouse over the number 10 Exit button. A yellow bar lights up.
 And when I move the mouse over the editor, at the scroll bar, when you click on the scroll bar of the editor,
  the Lcmd closes.
Saludos
Title: Re: LarsenCommander - new test version
Post by: Martin Iturbide on April 21, 2024, 06:02:32 pm
Hello Andi B.

Thanks for updating Larsen Commander.

I was wondering if there can be some easy way to make Larsen Commander to get the default font and size from the XWorkplace 1.0.26 font settings, without breaking the great functionality that Larsen Commander has to select a font and size for each component.

But I recognize I don't have  a clear idea of how I want it to be shown on the configuration. I don't know if having a checkbox on each of the components saying "Use system default", but it may look to messy.

Regards
Title: Re: LarsenCommander - new test version
Post by: Andi B. on April 21, 2024, 06:20:25 pm
Oh, I never used these setting page by myself. In fact I took the usual OS/2 way - drag/drop a font from the font palette to the desired place. I think I remember all these dropped fonts (and colors, and background colors) are saved and restored. So I think you only have to drop your desired fonts on the different controls and you're done. Does this not work for your on one of the controls?

Though I for myself rarely had the need to change my fonts the last decade or so as I still use monitors with 94-105dpi (most of the time).



Title: Re: LarsenCommander - new test version
Post by: Mentore on April 22, 2024, 08:26:44 am
I've just uploaded v1.08.02 at https://sourceforge.net/projects/lcmd-git/files/

From the Readme -
Code: [Select]
Change History:
---------------
20240416 v1.8.2
- Improve EA handling routines to reduce memory leaks and silent crashes
- Fix TRAP 000e in JFS while copying >69000 files. Hint, please use latest ArcaOS (kernel, JFS, ACPI 3.23.17, ...)
- Removed gl16os2.dll dependency (link .obj into .exe)
- Handel out of memory conditions more gracefully
- Added abtrace for logging via pmprintf.dll (if available on target system)
- Improved build cmd files to break on error (ERRORLEVEL handling)
- Tuneed lxlite options to not remove bldlevel info
- Change to David G. icon files (no clue where they are used)
- Again tuning of dynamic copy buffer size adoption (even more speed on my system :-) )
- Change icons to the ones supplied by David G. Thanks for that although it's not clear to me where they are used.
(- Disable integrated exception handlers in favor of exceptq) reverted back
- Disabled ExceptQ handler as I can't get it to work in conjunction with lcmds own handlers

Bug reports please here - https://sourceforge.net/p/lcmd-git/tickets/

Of course I'll keep reading here as well from time to time. Have fun.

Btw. I've a really hard time to track down and reduce this terrible JFS traps when copying a lot of files. It seems to me that JFS still does not touch some memory pages when feed with >4k buffers some times which I guess leads to these 000e traps. Combined with the memory leaks which LCMD suffered when lot of files where copied I had uncountable crashes with long chkdsks afterwards. Now the I reduced memory need for extended attribute handling I can't see these memory leaks anymore with above512. But on some scenarios LCMD still seems to eat up some system resources which report out of memory. Closing LCMD will recover from such situations now.

If you encounter hangs of LCMD you may look at the lcmd.log file in the directory of lcmd.exe. F.i. tail -n 50 lcmd.log will show you the last messages which usually shows some exception infos. For this you've to enable 'Debug' or 'Test' in Help - Debug - Options. Be aware running LCMD with 'Debug' for a long time or while copying huge amount of files let lcmd.log become very big in minutes.

Btw. if you've pmprintf.dll on your system PmPrintf.exe will show you these messages from LCMD with 'Debug' or 'Test' enabled too.

Downloading right now, I use LCMD all the time.
Should I notice some bug, I'll report it ASAP.
Thanks a lot Andi!

Mentore
Title: Re: LarsenCommander - new test version
Post by: Martin Iturbide on April 22, 2024, 06:33:59 pm
Hi
Does this not work for your on one of the controls?

Though I for myself rarely had the need to change my fonts the last decade or so as I still use monitors with 94-105dpi (most of the time).
Yes, it works for me with the usual way of changing the fonts with "Font Palette". But In my case I would like to adjust the font size a little bit bigger. I would like to have something that saves me some time grabbing by default font size (as XWP shows as default), but I understand that it is not easy to implement for each component described in the Font settings of lcmd.

Regards
Title: Re: LarsenCommander - new test version
Post by: Rich Walsh on April 22, 2024, 10:29:43 pm
I would like to have something that saves me some time grabbing by default font size (as XWP shows as default), but I understand that it is not easy to implement for each component described in the Font settings of lcmd.

It shouldn't be particularly hard: there are 5 entries under 'os2.ini->PM_SystemFonts'  [see below] and you map them to your categories. For example, 'PM_SystemFonts->Menus' maps to lcmd's 'AppWin.MenuBar' and its other menu settings.

DefaultFont: System Proportional - if this entry is missing/empty, assume 10 point SysPro

IconText:  WarpSans - this is mislabelled, it should be "Folder text" since it also is used for Details view
WindowText: WarpSans - standard dialog text

Menus: WarpSans Bold - both menubars and popups
WindowTitles:  WarpSans Bold

The key to using defaults while still allowing customization is *never* save the default value. That way you don't get stuck with a value that _used_ to be the default but isn't any longer.
Title: Re: LarsenCommander - new test version
Post by: Rich Walsh on April 23, 2024, 12:18:14 am
there are 5 entries [...] and you map them to your categories.

FWIW... you're not limited to just the 3 default fonts. Rather, you can use these entries as guidelines for setting the correct size of your preferred typeface. For example, if lcmd uses 'Helv' somewhere, you can get the entry for the category (say, "menu") but only pay attention to the size (use 'atol(...)). If 'Helv' has that point size, use it. Otherwise, go up or down at point _or_ substitute WarpSans (non-bold).

Do all this during init and save the results so you don't have to duplicate a lot of code. (Note: the default values in 'os2.ini' will never change during a session - they're only updated during shutdown - so there's no need to check for  new values.)
Title: Re: LarsenCommander - new test version
Post by: Andi B. on April 23, 2024, 04:56:30 pm
I understand the problem for people with high resolution screen usually found in modern Notebooks. But I think it should be possible with current version to set all controls to the needed bigger fonts. This setting should survive restarts. I'm I right Martin?

If yes I think it's not high priority to invest much time in this and so there's nothing which I can do in near future. F.i. the (rare) crashes and some leak in currently unknown system resources are more important IMO.

I just wait on Martins response and if there is no real show stopper (f.i. some controls can't be changed to usable fonts) I'll create a ticket with all the valuable infos from Rich. Maybe some time someone finds the time to make a more elegant solution for that.
Title: Re: LarsenCommander - new test version
Post by: Martin Iturbide on April 23, 2024, 09:10:17 pm
Hi Andi B.
I understand the problem for people with high resolution screen usually found in modern Notebooks. But I think it should be possible with current version to set all controls to the needed bigger fonts. This setting should survive restarts. I'm I right Martin?

If yes I think it's not high priority to invest much time in this and so there's nothing which I can do in near future. F.i. the (rare) crashes and some leak in currently unknown system resources are more important IMO.

I just wait on Martins response and if there is no real show stopper (f.i. some controls can't be changed to usable fonts) I'll create a ticket with all the valuable infos from Rich. Maybe some time someone finds the time to make a more elegant solution for that.
Like you said, this is not high priority. For me it is a "nice to have", it is not a bug. If there are no resources to do this, I think it is fine just to document it with a ticket.

Regards
Title: Re: LarsenCommander - new test version
Post by: Dariusz Piatkowski on April 28, 2024, 06:05:07 pm
Hi Andi,
...
File xfer performance wise, here are the results for movement of a test 800M AVI file:

1) from SAMBA share to ramdisk
OLD:  Peak=58,270K, Avg=43,550K
NEW: Peak=56,220K, Avg=46,516K

...so I'm thinking this is more about being network bound (1G LAN, but of course the Samba share box - NAS is the limitation here)

2) from ramdisk to SSD drive (Samsung 860 Evo, SATA)
OLD:  Peak= 82,788K, Avg=  45,545K
NEW: Peak=310,118K, Avg=220,910K
...

Repeat of the same performance tests for the newest version:

1) from SAMBA share to ramdisk
OLD: Peak=56,220K, Avg=46,516K
NEW: Peak=51,367K, Avg=45,832K

2) from ramdisk to SSD drive (Samsung 860 Evo, SATA)
OLD: Peak=310,118K, Avg=220,910K
NEW: Peak=503,901K, Avg=311,825K

...which naturally made my eyeballs nearly pop out b/c let's be honest, my SSD isn't that fast!!!

So a re-do produced the following NEW:
NEW: Peak=430,674K, Avg=183,007K

...which while still pretty high for a Peak, I suspect this has everything to do with my crazy ass JFS cache, which is 1G in size.

Couple more tests later though the higher Peak appears to be sustained, so perhaps there is a little bit more improvement here Andi!!!?

Thank you again!
Title: Re: LarsenCommander - new test version
Post by: Andi B. on April 28, 2024, 07:44:28 pm
Speed tests with SSDs vary on my system. I've tested with -
Samsung SSD 850 PRO 256GB
Samsung SSD 870 EVO 2TB
Samsung SSD 860 QVO 1TB
Crucial CT500MX500SSD1

It's sometimes not clear why copying from one SSD to another is faster with some setting while it's a bit slower with another target or source. I ever thought the maximum throughput you can get with an OS/2 system is copy from CLI cause a GUI program with progress indicator will always lower performance (the progress indicator needs to be update while CLI copy only waits for disk to accept new data, CLI does not even check CTRL-C). To my findings this is still true with disks although now LCMD comes very close to CLI. But in the last months I've even found scenarios where copying with LCMD is much faster than CLI copy (which TTBOMK uses 4k fixed buffer).

You may think copying with LCMD should be faster when not much else is running at the same time. But I've even scenarios where copying is actually a bit faster when LCMD lost the focus and some other heavy task are running. Once I even need a reboot to get the copy speed from/to a specific source/target to the level I'm used to. It was about 20% slower for unknown reason before.

Bottom line is it needs a lot of tests to get reproducible results. In most scenarios you can't get the same full speed with LCMD than with CLI but we are very close to that. Facts based on my tests for single big files are -
- Samba is way slower than ftp (more than 1,5 times slower here), both via netdrive *)
- for SSDs and harddisks LCMD is not exactly as fast als CLI copy but very close to it
- Ramdisk is way slower than SSD or harddisks (JFS is faster than HPFS but still 3 times slower)
- Peak values may be very high cause of SSD (and JFS) cache

*) Samba is way faster than FTP with a lot of small files I think because of it's directory caching which FTP is missing
Title: Re: LarsenCommander - new test version
Post by: Andi B. on February 23, 2025, 04:22:34 pm
I've just released v1.09.00. See https://sourceforge.net/projects/lcmd-git/files/

For me this is a major improvement as now the EA logic finally works without memory leaks. At least for my (extensive) testing. Also the focus switching problem improved a lot. Maybe not fully solved but at least the test cases I had. The copy buffer size limit setting now will be honored again. Mind now in MBytes not KBytes. Not sure if translations in all languages are correct.

Unfortunately there is still one problem I couldn't solve. But during extensive testing and debugging I'm pretty sure this is a file system driver (JFS) problem. Copying huge amount of files can lead to file system hangs. Sometimes after 80'000 files, sometimes after 800'000 files. First I thought it's a problem with log file overflow (beyond 2GB). But now as I changed this to LFS support too and numerous test with and without logging file enabled I'm pretty sure it's something very low level in JFS or NVME or ? driver. Given that no one else mentioned this problem until now and given 1.09 is able to copy at least 10 times more files without problems than any version before I think it's time to make it public. Enjoy v1.09.00.

Code: [Select]
Change History:
---------------
20250222 v1.9.0
- Improved Extended Attribute handling again. Now it should handle all sizes of EAs and checks and limits to <64k
- EA handling now really fixes memory leaks
- Reenabled max copy buffer size setting in but now use MB instead of KB because it is in HigherMem arena since
1.8.0 (?) anyway
- Improved focus switching logic. Don not set focus on unrelated windows after copy/del operations.
- Skip sending lot of GUI (PM) messages for updating progress dialog with fast disks while copying and deleting files.
This is to work around PM memory leaks (crashes) when hammered with very much messages in a short period of time.
With fast disks copying or deleting small files takes few ms or even less. The previous logic to update the
progress bar after each file (and during a file when it is bigger) is changed to not send more than about one
message in 125ms. PM seems to be not safely ignore to much unhandled messages and starts to eat up shared memory.
Eventually this crashes the whole system when the process is not closed before. F.i. this skipps more than 69000
messages when copying the ApacheOpenOffice source code tree. The same skipping logic improves speed when deleting lot
of files. F.e. 33s instead 53s for AOO source code tree (about 70k files). More than 76000 messages where skipped with
that new delete logic.
- Change LOG file handling to LargeFileSupport to allow more than 2GB log files. This is needed cause with very heavy
logging previous versions hang the whole file system starting with the log file directory at 2GB log file size
(> 100k files copied with DEBUG)
Title: Re: LarsenCommander - new test version
Post by: Mentore on February 24, 2025, 08:52:54 am
I've just released v1.09.00. See https://sourceforge.net/projects/lcmd-git/files/


Hi Andi, I'm about to test it  right now. I'm heavily using LC, since I'm so used to the old Norton Commander interface. I also would like to ask you if it's possible to add an "Open With" option on files.
Since I use it really much when I develop or port OS/2 software, something like that would be of great help in order to let me, say, open a source code file with an appropriate editor like VisualSlickEdit, EPM, plain old E, FTE, or maybe a more complex IDE like VisualAge or OpenWatcom.
Was that already on your mind?
In any case, thanks for your good work on this file manager, I feel really at home using it.

Mentore
Title: Re: LarsenCommander - new test version
Post by: TeLLie on February 24, 2025, 03:12:42 pm
Hi Andi,

Thankz for this new version :)

greetz TeLLie

Title: Re: LarsenCommander - new test version
Post by: Lars on February 25, 2025, 10:50:33 am
I've just released v1.09.00. See https://sourceforge.net/projects/lcmd-git/files/

For me this is a major improvement as now the EA logic finally works without memory leaks. At least for my (extensive) testing. Also the focus switching problem improved a lot. Maybe not fully solved but at least the test cases I had. The copy buffer size limit setting now will be honored again. Mind now in MBytes not KBytes. Not sure if translations in all languages are correct.

Unfortunately there is still one problem I couldn't solve. But during extensive testing and debugging I'm pretty sure this is a file system driver (JFS) problem. Copying huge amount of files can lead to file system hangs. Sometimes after 80'000 files, sometimes after 800'000 files. First I thought it's a problem with log file overflow (beyond 2GB). But now as I changed this to LFS support too and numerous test with and without logging file enabled I'm pretty sure it's something very low level in JFS or NVME or ? driver. Given that no one else mentioned this problem until now and given 1.09 is able to copy at least 10 times more files without problems than any version before I think it's time to make it public. Enjoy v1.09.00.

I would suspect this to be a JFS problem: https://github.com/bitwiseworks/libcx/issues/36

Unfortunately, I think "breaking up a large transfer" would mean that the copying application has to do it (unless you are using libcx and the gcc compiler and not calling the OS/2 API directly). If possible and as a test case, you could create a HPFS partition and copy from that and see if that avoids the problem.
Title: Re: LarsenCommander - new test version
Post by: Andi B. on February 25, 2025, 11:43:27 am
Interesting. But if I understand it correctly it should be fixed since a while. I can reproduce problems here even with ArcaOS5.1EN.

OTOH LarsenCommander doing the copy stuff a bit unusual so this may be another JFS problem others rarely encounter. Lcmd reads the file (in chunks if bigger than internal dynamic buffer) and writes it to target. So far so good. Then it reads in all EAs from source file and then writes the EAs to the target. This EA handling is strange when you see it the first time. Reading every EA name and data with dynamic size.... And I think the problem occurs when writting the EAs.

Unfortunately I didn't find a way on OS/2 to read and write all EAs of a file at once. These can be <64k anyway so no would be no problem on todays systems. Reading all these EA entries in a loop by name and values is a mess.
Title: Re: LarsenCommander - new test version
Post by: Lars on February 26, 2025, 12:39:12 am
Quote
Interesting. But if I understand it correctly it should be fixed since a while. I can reproduce problems here even with ArcaOS5.1EN.
A workaround is in place if you use "read" and "write" Posix functions as these break up large transfers into multiple smaller transfers before they eventually invoke DosRead (DosWrite). The native OS/2 API call "DosRead" continues to suffer from JFS quirks and limitations. It's the IFS driver that is broken, not the DosRead API function.


EAs: DosQueryFileInfo, DosQueryPathInfo, DosSetFileInfo, DosSetPathInfo, "pInfo" pointing to EAOP2 structure:

Getting all EAS at once is done with info level = FIL_QUERYALLEAS = 4.
fpFEA2LIST pointer will point to list of all EAs queried, including their values. You will need to hop from one entry to the next following the oNextEntryOffset field (the individual FEA2s are varying in length). Provide a sufficiently large buffer in fpFEA2List (> 64k, better twice that size) so that it can be filled.

Setting all EAS at once is done with info level = FIL_SETEASFROMLIST = 2.
fpFEA2LIST pointer points to list of all EAS to set, including the values to set. Need to set the oNextEntryOffset to point to the next element (the individual FEA2s are varying in length). Provide a sufficiently large buffer in fpFEA2List (holding all your EAs). You have to ensure that you do not exceed maximum EA overall size.

These info levels are undocumented in the CPS.

You should be able to feed the result of FIL_QUERYALLEAS (read EAs with DosQueryXXXXInfo) into FIL_SETEASFROMLIST (write EAs with DosSetXXXXInfo).

DosEnumAttribute will give you a list of EAs (names) that exist. It's not really needed if you just make the EA buffer large enough (we know the maximum but take into account EA structure and EA name lengths, not only values).
Title: Re: LarsenCommander - new test version
Post by: Andi B. on February 26, 2025, 08:37:02 am
Quote
These info levels are undocumented in the CPS.
Wow. Some of your explanations sounds familiar. After I invested countless hours on reading original Lcmd code, edm2 and all the help files of os2tk45. But where did you get the undocumented stuff from? Answers from some experts in some goups which you noted somewhere? Or is there a place where someone document such undocumented stuff? Stevens site?

Btw: these idiots which wrote the control programming appendum for DosOpenL f.i. should be sent to jail. Again I stumbled over this wrong description and examples (variables versus pointers). It's a mess when the official documentation is so wrong in so many places :-(.

Btw2: It's a somehow educated guess that the file system is faulty here. With different versions of EA handling in Lcmd I very often got 'Trap 000e in JFS'. The last implementation which I think perfectly correct, let the IFS silently stop working only sometimes. Unfortunately I need to copy a few hundred thousand files with a lot of GBs. So still have no simple scenario to reproduce.

Btw3: For my problem I think it's not the DosRead buffer size as this is already limited (<1 - ~16MB) here. AFAIK the problem Silvan reported happens with about 400MB and above. OTOH David explained in the ticket it's not the buffer size which triggers this one known problem, but 2 DosOpenL calls where the first one does nothing. Do you think Davids explanation in the ticket is wrong? Or is he talking about another problem? Is it worth to test with read buffers max. 1MB f.i.?
Title: Re: LarsenCommander - new test version
Post by: Lars on February 26, 2025, 09:57:29 am
Hi Andi,

Quote
These info levels are undocumented in the CPS.
Wow. Some of your explanations sounds familiar. After I invested countless hours on reading original Lcmd code, edm2 and all the help files of os2tk45. But where did you get the undocumented stuff from? Answers from some experts in some goups which you noted somewhere? Or is there a place where someone document such undocumented stuff? Stevens site?

...

Btw3: For my problem I think it's not the DosRead buffer size as this is already limited (<1 - ~16MB) here. AFAIK the problem Silvan reported happens with about 400MB and above. OTOH David explained in the ticket it's not the buffer size which triggers this one known problem, but 2 DosOpenL calls where the first one does nothing. Do you think Davids explanation in the ticket is wrong? Or is he talking about another problem? Is it worth to test with read buffers max. 1MB f.i.?

For EAs: I looked into the OpenJFS Source code at netlabs.org and what it does. Needless to say that it does stuff that is not documented but I took it by the word (and hope HPFS does the same):

https://trac.netlabs.org/OpenJfs

and I looked at a very old OS/2 Version 1.2 IFS Specification written by Microsoft in 1989 that I found in the internet, it appears as part of European patent EP0415346A2:

https://patentimages.storage.googleapis.com/23/69/45/f6a4cb501938a9/EP0415346A2.pdf

it was scanned by someone from a pile of paper (and that I cleaned from junk and reprocessed as a new PDF to make it more accessible). Don't forget to scroll through it all the way !
I have no clue of why some of the info in the old IFS specification got lost along the way but on the other hand it covers stuff that is no longer relevant/workable, you need to be careful ...

In the old IFS spec, you can read the following for DosQFileInfo / DosQPathInfo:

Quote
*** THE FOLLOWING UNPUBLISHED FUNCTION WILL GO AWAY IN FUTURE *****
*** RELEASES AND SHOULD BE UNPUBLISHED ***
Level 0x0004 file information returns all EA info for the file. On input, FileInfoBuf is an EAOP structure above. fpGEAList contents are ignored. fpFEAList points to a data area where the relevant FEA list will be returned. The length field of this FEA list is valid, giving the size of the FEA list buffer. offError is ignored.
...
Guess what, that info level never went away (translate "DosQueryFileInfo" for "DosQFileInfo"), likely because some app used it and then MS needed to continue supporting it :-)

Likewise for DosSetFileInfo / DosSetPathInfo:

Quote
Level 0x0002 file information sets a series of EA name/value pairs. On input, PathInfoBuf is an EAOP structure above. fpGEAList is ignored. fpFEAList ponts to a data area where the relevant FEA list is to be found. offError is ignored.

I guess that by now, you get the idea :-)


For AN ticket: I no longer have access to the AN ticket system and therefore cannot see what they found out.



Title: Re: LarsenCommander - new test version
Post by: Andi B. on February 26, 2025, 12:21:27 pm
Thank you very much. Enough stuff to test and try out. Have to bookmark this. OTOH I thought this lcmd version would be really the last one.....

Ouch, I don't have access to the source code at netlabs :-(.

Title: Re: LarsenCommander - new test version
Post by: Martin Iturbide on February 26, 2025, 01:20:11 pm
Ouch, I don't have access to the source code at netlabs :-(.
It this one? https://trac.netlabs.org/openjfs/browser#branches
I can watch it without being logged on. Maybe it was the upppercase on the URL.

Regards
Title: Re: LarsenCommander - new test version
Post by: Andi B. on February 26, 2025, 01:54:30 pm
Ouch, I don't have access to the source code at netlabs :-(.
It this one? https://trac.netlabs.org/openjfs/browser#branches
I can watch it without being logged on. Maybe it was the upppercase on the URL.

Regards

I was at the main trac page and even logged in and did not found the usual 'source code' button. Just found out netlabs has 2 jfs projects. One jfs another openjfs. I was here - https://trac.netlabs.org/jfs while correct link should be - https://trac.netlabs.org/openjfs/.

Last check in seems to were 21 years ago. Not sure how much of this old code is still in use. Thanks anyway.
Title: Re: LarsenCommander - new test version
Post by: Martin Iturbide on February 26, 2025, 05:23:14 pm
Hi
I was at the main trac page and even logged in and did not found the usual 'source code' button. Just found out netlabs has 2 jfs projects. One jfs another openjfs. I was here - https://trac.netlabs.org/jfs while correct link should be - https://trac.netlabs.org/openjfs/.

Last check in seems to were 21 years ago. Not sure how much of this old code is still in use. Thanks anyway.

I get it now.
- OpenJFS is based in GPL code released by IBM. IBM released JFS under the GPL license to be merge with Linux.
- JFS. Is the IBM driver for OS/2 and it is evolution that is under... I have no idea.. possible under the IBM DDK license that is incompatible with open source. IBM DDK license allows you to create a freeware driver binary, but does not allow you to publicly share the source code. What can be done with this license (as an alternative) it to have a collaborative close source project with one organization (like Netlabs) taking the ownership of the changes, and every contributor should agree to the IBM DDK.

Sorry of the long license story.

Regards
Title: Re: LarsenCommander - new test version
Post by: Dave Yeo on February 26, 2025, 05:40:15 pm
There's also https://github.com/OS2World/DRV-JFS-GPL (https://github.com/OS2World/DRV-JFS-GPL) which is the 1st IBM release of the GPL code and contains the OS/2 branch. Likely OpenJFS is based on this.
Title: Re: LarsenCommander - new test version
Post by: Lars on February 26, 2025, 07:58:10 pm
Here is code to prove it works at least with JFS (I have all my files on JFS partitions):

Code: [Select]
#define INCL_BASE
#include <os2.h>
#include <stdio.h>

#define FIL_QUERYALLEAS 4

#define BUFSIZE 1024UL * 128


int main(int argc,char *argv[])
{
   EAOP2 eaop={0};
   PFEA2 pFEA = NULL;
   ULONG oNextEntryOffset;
   APIRET rc = NO_ERROR;
   ULONG i=0;
   PUCHAR pHexValueByte = NULL;

   if (argc != 2) {
      return 1;
   } /* endif */

  rc = DosAllocMem((PPVOID)&eaop.fpFEA2List,BUFSIZE,PAG_COMMIT | PAG_READ | PAG_WRITE);

  if (NO_ERROR == rc) {
     eaop.fpFEA2List->cbList = BUFSIZE;
     rc = DosQueryPathInfo(argv[1],FIL_QUERYALLEAS,&eaop,sizeof(eaop));
     if (NO_ERROR == rc && eaop.fpFEA2List->cbList > sizeof(eaop.fpFEA2List->cbList)) {
        pFEA = eaop.fpFEA2List->list;
        do {
           oNextEntryOffset = pFEA->oNextEntryOffset;
           printf("Name:%s,Value:",pFEA->szName);
           pHexValueByte = pFEA->szName+sizeof(UCHAR)+pFEA->cbName;
           for (i=0;i<pFEA->cbValue;i++,pHexValueByte++ ) {
              printf("%.2x ",*pHexValueByte);
           } /* endfor */
           printf("\n");
           pFEA = (PFEA2)((PCHAR)pFEA + oNextEntryOffset);
        } while (oNextEntryOffset  ); /* enddo */
     } /* endif */
     rc = DosFreeMem(eaop.fpFEA2List);
  } /* endif */

  return 0;
}

Pass it a filename and it will print out the EA info as hex bytes (not awfully helpful, obviously but you can crosscheck with what the WPS displays).
Title: Re: LarsenCommander - new test version
Post by: Dariusz Piatkowski on April 05, 2025, 10:39:33 pm
Hello Andi,

Sorry about the long delay in providing my feedback!

I've just released v1.09.00. See https://sourceforge.net/projects/lcmd-git/files/

Code: [Select]
Change History:
---------------
20250222 v1.9.0
- Skip sending lot of GUI (PM) messages for updating progress dialog with fast disks while copying and deleting files.
This is to work around PM memory leaks (crashes) when hammered with very much messages in a short period of time.
With fast disks copying or deleting small files takes few ms or even less. The previous logic to update the
progress bar after each file (and during a file when it is bigger) is changed to not send more than about one
message in 125ms. PM seems to be not safely ignore to much unhandled messages and starts to eat up shared memory.
Eventually this crashes the whole system when the process is not closed before. F.i. this skipps more than 69000
messages when copying the ApacheOpenOffice source code tree. The same skipping logic improves speed when deleting lot
of files. F.e. 33s instead 53s for AOO source code tree (about 70k files). More than 76000 messages where skipped with
that new delete logic.

Alright...this is by far my favourite file manager, so I do put it through it's fair share of paces.

Everything seems to be alright with the exception of the fact that the 'Bytes per second' window no longer shows the graphical throughput rate as the file copy progresses.

I repeated my file copy/move several times (between different SOURCE=>TARGET combinations and this is consistent.

So I left the above section of your post purposely because I now wonder if perhaps the "...Skip sending lot of GUI (PM) messages..." is the root cause of this???

Othwerise, things are good...thank you again for maintaning this !!!
Title: Re: LarsenCommander - new test version
Post by: Andi B. on April 13, 2025, 11:47:39 am
Quote
Everything seems to be alright with the exception of the fact that the 'Bytes per second' window no longer shows the graphical throughput rate as the file copy progresses.

Good catch. I wonder why I've overlooked this. Needs investigation.