LarsenCommander - new test version

[Andi B.]

Since a while I'm working on some problems in LarsenCommander again. There is still one problem I want to resolve before I make an official release. But as I'm preparing for holiday vacation I think it may be good to spread the current version to the public.

Main difference to older releases is - I reenabled big buffers for file copy operations. The advantage is much faster copy speed especially with big files. Copy speed is still not as fast as the copy command from a 4os2 command prompt but the difference is not as dramatic than before. I had to rework and extend the original dynamic buffer adaption code for this. It seems current copy speed is the maximum you can get when copying with an PM application.

The max. copy buffer size in old releases was determined by a simple algorithm and based on the setting in Options - Various - File Copy/move tab. I had to limit the buffer in some 1.7. release to 60k cause there was a subtle bug in ndfs DAV plugin which leads to some zero bytes in large files copied with the help of big buffers. The side effect was a limited copy speed at about 30MBytes/s I just discovered a few months ago. There's an old ticket about the ndpdav bug which is resolved in newer releases I think. So I think it's save to reenable big buffers again. For convenience I include my fixed ndpdav.dll in this lcmd*zip. But you probably don't need it. Current ndfs 3.1.6 also has changed in a way that such subtle plugin bugs shouldn't show up anymore. But be warned if you use old ndfs/ndpdav versions.

Lcmd also uses high memory for the copy buffer now. So if you watch your memory consumption (f.i. with above512.cmd or theseus) you may find a bit less memory usage from the scarce lower shared mem. Unfortunately this does not fix another long standing problem - copying big trees of directories sometimes leads to eat up a lot (sometimes all) lower shared memory. This may eventually bring your system down until you kill lcmd. Finally I can reproduce this problem but unfortunately I don't have a fix for this right now. If you ever run into this bug in the past, sorry there's no fix in this release for that.

In this archive I've also included the current available translations. When you want to use lcmd with another language or make your own translation look into the locale directory.

I've also checked in all current sources at the official home of LarsenCommander - https://sourceforge.net/projects/lcmd/

Have fun :-)

[Martin Iturbide]

Hi Andi B.

Thanks for taking the time to update Larsen Commander.

I was trying the new version and I have a little question or issue. When I have a file on the right side, and I want to copy it to the left side (Different folder). I select the file and click "copy", a popup shows telling me that it will copy the file from here to there, I click ok, the file get copied but the desktop windows loose focus, the focus moves around and a window from the background gets to the top. (Maybe the pictures explain my problem if my wording is not clear enough)

It also happens when I delete a file with the right click menu too.  Is that only happening to me? 

Regards

[Andi B.]

Maybe you're bitten by the same bug I mentioned in the ToDo.txt file more than 12 years ago ;-) (20100810 After deleting files focus changes to another not lcmd window).

I remember I added some debug messages back then when I tried to track this down. But couldn't solve this. But as this is a nuisance for me too chances are high that I finally will fix it.

 

[Andy Willis]

Thank you Andi.
I updated my local svn and built it.  Thought I had a problem with the executable at first, I tried to test the built lcmd.exe in the directory it was built in and it threw errors but I figured out it was reading the lcmd.rc for building it and the executable needs its own lcmd.rc file.  Once I worked that out, it seems to be working well.
One thing that I would like I would like to see (maybe I can work it out) is to have lcmd.exe look for gl16os2.dll in its own directory first before looking at libpath, this would allow it to run without gl16os2.dll having to be in libpath.

[Andi B.]

this would allow it to run without gl16os2.dll having to be in libpath.

I don't think I've it in the libpath. But I will check next week.

[Dariusz Piatkowski]

Hello Andi!

As the others have stated: big Thank You!!!

...Main difference to older releases is - I reenabled big buffers for file copy operations...

The max. copy buffer size in old releases was determined by a simple algorithm and based on the setting in Options - Various - File Copy/move tab. I had to limit the buffer in some 1.7. release to 60k cause there was a subtle bug in ndfs DAV plugin which leads to some zero bytes in large files copied with the help of big buffers....

So I'm curious about this: does the current Max Buffer setting still function, or is that now ignored by your enhanced logic?

I had previously set this to 10K, now set to 64K, tested across the local as well as LAN devices (Win boxes as well as a NAS), all still good.

OK, so I do have a 'feature request': WHEN looking at the contents of a ZIP file do you think you might be able to enable object recognition and the use of system-wide settings instead of the default editor that's defined in LCMD settings?

I'm literally thinking:
1) TEXT is fine
2) JPG, GIF, etc...do not work
3) MP4, AVI...do not work

...so am I perhaps missing something here, or is LCMD really limitted to just being able to treat everything inside a ZIP  file as a TEXT file?

Thanks!

EDIT
====
I clarified that we are talking about ZIP file contents here...my original post did NOT spell this out at all!

[Andi B.]

Thank you Andi.
I updated my local svn and built it.  Thought I had a problem with the executable at first, I tried to test the built lcmd.exe in the directory it was built in and it threw errors but I figured out it was reading the lcmd.rc for building it and the executable needs its own lcmd.rc file.  Once I worked that out, it seems to be working well.

Yes I was confused by this more than once too. There's included the lcmd_text.cmd (and te.cmd) which calls

Code: [Select]

start lcmd.exe -rcFile ..\..\lcmd.rc -rcSearchPath ..\..\src\glib;..\.. -iniFile lcmd.ini -logFilter test -logFile lcmd.log %1 %2 %3 ...

One thing that I would like I would like to see (maybe I can work it out) is to have lcmd.exe look for gl16os2.dll in its own directory first before looking at libpath, this would allow it to run without gl16os2.dll having to be in libpath.

I've ';.;' in libpath after the %unixroot% directories.

Code: [Select]

LIBPATH=p:\usr\local\lib;p:\usr\lib;p:\dll;.;P:\.....So that's no problem for me. Isn't that the usual setup? I mean having ;.; in the libpath (best at the end or after unixroot dirs. Isn't it the standard behavior for OS/2 programs to search their dlls in libpath? Or start it by an object set to run in the programs directory?

Are you asking for some 'special' code in LCMD which loads the dll from the programs directory regardless of libpath setting? Are there some example programs for such?

[Andi B.]

....
So I'm curious about this: does the current Max Buffer setting still function, or is that now ignored by your enhanced logic?

Currently it's only honored when you set it bigger than 16MByte. And only for the first file block which is copied. If the file is bigger than 16MB (or your setting) then the buffer may be increased when copying one block takes less than 40ms. And the buffer size gets decreased when copying a block takes more than 350ms. Actually the 'ideal' copy buffer size calculation is a bit more complex but the idea is to change the buffer size dynamically so that copying one block takes between about 40 - 350ms. This all has a big impact on real big files (lot of 100MBytes) and fast devices (f.i. SSDs or fast HDDs).

Originally the 'ideal' buffer size was calculated for a block transfer time of 4s (if I understand the code correct) but limited to your setting in the options and a rather small initial setting. I had to change that cause some overflow in the calculation leads to wrong buffer size and very slow transfer rates in some cases. Moreover I think most users didn't even find the setting in the options (like myself) and so the got only low transfer rates even with very fast devices.

With my current algorithm I have about 85MBytes/s when writing to HDDs and >140MBytes/s on some SSD. This is close enough to the limit my system is able to handle and I think most I can get out of a PM copy program with progress indication.

I mainly tried to optimize copy speed with big files and reals disks within the system. I've not seen a negative impact in copy speed for small files. Limited testing was done on different LAN or USB devices and different file systems (JFS, HPFS, FAT, FAT32) too without negative side effect. Unfortunately I've currently no chance to test fast USB3 devices. They do not work well on my system in general. So test result with different settings are welcome. Mind you always can compare speed with copy done from cli. Usually copy from cli is still faster. But not that much anymore than before.

...
OK, so I do have a 'feature request': WHEN looking at the contents of a ZIP file do you think you might be able to enable object recognition and the use of system-wide settings instead of the default editor that's defined in LCMD settings?
...

I consider the vfs implementation (zip file handling) in LCMD as a proof of concept. Only basic functionality is implemented AFAIK. As you already have observed functionality is very limited. I never used this feature by myself. I find even file managers which implement such feature much better (TotalCommander on Win) as rather useless and I don't use it. Cause even there it never works nearly as good as a real Archive program manager (WinRar, 7-zip, Win folder zip integration, ...). I don't see any real chance for me to implement zip/rar/arj/tar/iso/... file handling in LCMD to an extend I by myself would call it nearly useful. Not even if I got fired by my company and could work full time on this ;-). So until no one else could invest a lot of time in this part of LCMD I would suggest setting 'System launch Zip's by default'.

[Andy Willis]

Thank you Andi.
I updated my local svn and built it.  Thought I had a problem with the executable at first, I tried to test the built lcmd.exe in the directory it was built in and it threw errors but I figured out it was reading the lcmd.rc for building it and the executable needs its own lcmd.rc file.  Once I worked that out, it seems to be working well.

Yes I was confused by this more than once too. There's included the lcmd_text.cmd (and te.cmd) which calls

Code: [Select]

start lcmd.exe -rcFile ..\..\lcmd.rc -rcSearchPath ..\..\src\glib;..\.. -iniFile lcmd.ini -logFilter test -logFile lcmd.log %1 %2 %3 ...

One thing that I would like I would like to see (maybe I can work it out) is to have lcmd.exe look for gl16os2.dll in its own directory first before looking at libpath, this would allow it to run without gl16os2.dll having to be in libpath.

I've ';.;' in libpath after the %unixroot% directories.

Code: [Select]

LIBPATH=p:\usr\local\lib;p:\usr\lib;p:\dll;.;P:\.....So that's no problem for me. Isn't that the usual setup? I mean having ;.; in the libpath (best at the end or after unixroot dirs. Isn't it the standard behavior for OS/2 programs to search their dlls in libpath? Or start it by an object set to run in the programs directory?

Are you asking for some 'special' code in LCMD which loads the dll from the programs directory regardless of libpath setting? Are there some example programs for such?

I have lcmd in path but am trying to avoid having to put the dll in the libpath.  There is a way when calling the DLL to search the path the the executable is in first, ;.; only helps if you are in the directory that the DLL is in when you launch the executable.  I should be able to find the code in Mozilla.

[Andi B.]

I have lcmd in path but am trying to avoid having to put the dll in the libpath.  There is a way when calling the DLL to search the path the the executable is in first, ;.; only helps if you are in the directory that the DLL is in when you launch the executable.  I should be able to find the code in Mozilla.
[/quote]
Ok now I understand. Different people, different habits. I don't have the LCMD directory in PATH nor in LIBPATH. I've three different objects in xcenter which starts LCMD with different settings and at least one object on the desktop. Looking at it I now see one of them points to te.cmd. Others have '*' as 'Path and file name' and something like '/c set beginlibpath="P:\Util\Larsen Commander_OwnTest" && "P:\Util\Larsen Commander_OwnTest\lcmd.exe" -rcSearchPath P:\Util\Larsen Commander_OwnTest -rightDir E:\_work -leftDir E:\Dokumente -iniFile E:\home\lcm_own.ini' as Parameters. Now I see why I don't need it to bee in any (LIB)PATH :-)

If I have a need to start a program from cli then I usually have a *.cmd file in my default \batch file directory (which is in PATH). With this habit (objects in xcenters and .cmd files) for a lot of programs I keep my PATH and LIBPATH short. As said different users different needs.

Anyway if you find some solution for your problem maybe we can implement it.

[Rich Walsh]

One thing that I would like I would like to see (maybe I can work it out) is to have lcmd.exe look for gl16os2.dll in its own directory first before looking at libpath, this would allow it to run without gl16os2.dll having to be in libpath.

Are you asking for some 'special' code in LCMD which loads the dll from the programs directory regardless of libpath setting? Are there some example programs for such?

The solution to this particular problem is to get rid of the DLL -- there's no reaon for it to exist. The original author claimed that VACPP 4.0 couldn't handle a 16-bit call so he created a separately-compiled file. But rather than simply linking it into the exe with all the other .obj files, he created a dll. Why?

VAC 3.65 has no problem with 16-bit calls, and AIUI, 3.65 is just 4.0 without the class libraries. I'd expect you could compile this file the same as all the others, then add it to the ilink cmd line. Even if you do need to use a different compiler, it should still link OK.

[Lars]

I think, this misunderstanding stems from the fact that every IOPL example (which means 16-bit code use by the nature of IOPL support in OS/2) seems to imply that you need to put the IOPL code into a separate DLL. Well, you do not have to do that, just needs some additions to the def file and you can happily link 16-bit IOPL code to the rest of your 32-bit app.

[Andi B.]

Thanks Rich and Lars. I'm thinking about your inputs and hope to find the time to come back to this.

Rich, to my understanding VAC4 is a complete different thing. It uses vacbld which creates a code store and compiles all sources at once (?) instead classic icc for every file and makefiles. But in the package of VAC4 there's also included v3.65 with a lot of information when you can or should switch from older versions to 4.0 or to 3.65. I read them long ago and probably forgot most of them. Anyway I'm thinking about switching to gcc since a while. But it's not that easy and so not top priority.

Currently I try to track down a memory leak in LCMD in some situations. Doing this I stumbled over a shortcoming in pmprintf which eats up all lower shared memory when loaded very heavy. So I try to improve pmprintf first (make use of high mem). While doing this I was bitten by a shortcoming of our os2ahci driver. Reproducing this problem, creating a ticket and help to track it down took a lot of time. So it's all about having to less time for all these interesting stuff to get finally solved. So better not giving any time frame....

[Andi B.]

I've upload a new version here - https://sourceforge.net/projects/lcmd-git/files/

Mind this is a new project page at sourceforge as I'm using git instead svn now.

This is a lxlite stripped down debug version with the debug information in the *.dbg file. In case of an exception exceptq should be able to create a nice report. At least when the lcmd build in exception handlers will not catch it before (ToDo for future releases). Although debug symbols are compiled in, pmprintf.dll should not be needed for this build.

Changes from last release are -
- Fine tuning for dynamic copy buffer size adoption
- Reduce logging while copying huge amount of files (reduces memory needs and log file size)
- Disable playing priority games in hope of better overall system response
- Adding ExceptQ handler (not working yet as own exception is still active)
- Fixed some cppcheck warnings (thanks Andy Willis for that)
- Add some comments for using highmem in some places (currently not used)
- Add writing to PmPrintf queue in parallel to log file when #ifdef __DEBUG_TRACE__
- Use HighMem for copy buffer

Especially the tuning of the copy buffer algorithm makes a big difference in copy speed in so some cases. At least with my rather fast HDDs and SSDs and big files. But also with Samba shares or other source/targets differences can be observed. All my measurements gave good results. I think this is near to the maximum copy speed we can reach with an PM OS/2 application with progress bar. If you think copy speed now is slower than with any previous version please share your experience here with clear instructions to reproduce. I really tested a lot scenarios with different targets and think the current implementation is better then everything before. Mind 'copy' from command line is most of the time still a bit faster as it does not have the overhead of a graphical progress bar and no checks for interruption during copying a file.

Get ride of gl16os2.dll is still on the ToDo list though.

[Dariusz Piatkowski]

Hi Andi,

I've upload a new version here - https://sourceforge.net/projects/lcmd-git/files/
...
Changes from last release are -
- Fine tuning for dynamic copy buffer size adoption
- Reduce logging while copying huge amount of files (reduces memory needs and log file size)
- Disable playing priority games in hope of better overall system response
- Adding ExceptQ handler (not working yet as own exception is still active)
- Fixed some cppcheck warnings (thanks Andy Willis for that)
- Add some comments for using highmem in some places (currently not used)
- Add writing to PmPrintf queue in parallel to log file when #ifdef __DEBUG_TRACE__
- Use HighMem for copy buffer

Especially the tuning of the copy buffer algorithm makes a big difference in copy speed in so some cases. At least with my rather fast HDDs and SSDs and big files. But also with Samba shares or other source/targets differences can be observed. All my measurements gave good results...

Application functionality seems to be spot-on.

File xfer performance wise, here are the results for movement of a test 800M AVI file:

1) from SAMBA share to ramdisk
OLD:  Peak=58,270K, Avg=43,550K
NEW: Peak=56,220K, Avg=46,516K

...so I'm thinking this is more about being network bound (1G LAN, but of course the Samba share box - NAS is the limitation here)

2) from ramdisk to SSD drive (Samsung 860 Evo, SATA)
OLD:  Peak= 82,788K, Avg=  45,545K
NEW: Peak=310,118K, Avg=220,910K

...here I'm thinking there is an OBVIOUS and huge difference! In fact I was floored by how much of a change this was, pretty much maxing out the throughput I would normally see on my DISKIO tests, so I'd say this is an extremely good result Andi!!! Cheers to you sir.

Now, a couple of further remarks as I'm wondering if you would consider looking into these further:

1) NOT an obivous bug, but maybe one???
Opening up a MP4 file causes my MPLAYER app to be started, but because I actually have a CMD that drives it (which accepts a parameter - [Input Scaling Factor] "%*" - that pops up as that INPUT box which allows me to input a scaling factor) the video starts playing, but I notice that the MPLAYER app actually was started twice.

This is why I'm thinking it maybe a bug...and perhaps only dealing with instantiating CMD started apps, because doing the very same thing for PDF files for example opens up a single instance of Lucide.

2) file type specific icons
Let me use the example of JPGs. If these have the icon stored as EA, LCMD pulls these in and shows them. However my NAS does not support EAs, and therefore all of the pics simply show a document icon as opposed to the DEFAULT PMVIEW icon.

This behavious applies to numerous other filetypes as well.

Would it therefore be possible to modify LCMD to use default APP-specific ICONS for particular filetype when the EAs do not have the icon information available?

Once again, thank you for maintaining and enhancing LCMD!!!