OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Applications => Topic started by: Remy on April 25, 2020, 09:37:40 pm

Title: Shared storage
Post by: Remy on April 25, 2020, 09:37:40 pm
Hi !

Any idea why an older ffmpeg compiled with gcc 4.4.4 doesn't left any shared storage holes at close time while ffmpeg compiled with gcc 5.5.0 left about 20MB (all dlls size) of free unusable holes in the shared memory at close time ?

(FFmpeg or ffprobe)

Updt: problem occures as well with ffplay but in all cases only with dll's loaded above 512MB line  (when below, no issue)


Regards
Rémy
Title: Re: Shared storage
Post by: Remy on April 27, 2020, 04:36:28 am
Well, same raison as for firefox for which ffturbo was build.

I tested some rexx functions which seems to well work.
Working with this now.

Cheers/2
Title: Re: Shared storage
Post by: Remy on April 27, 2020, 09:24:17 am
Problem resolved and it works pretty well.

I coded a small rex which loads ffmpeg dlls (set to be high loaded by ABOVE512 or highmem) permanently into high shared storage. (all ffmpeg 4.0.2 eligible dlls are coded in the rexx. It work like ffturbo (without the need to have a running background process) but ffturbo is more complex due to libpathstrict...

If you are interested by my rexx code, send me an email from under my web site, same as AVxCAT, CLAMAVGUI...
remydodin*
Title: Re: Shared storage
Post by: Remy on April 28, 2020, 07:49:29 pm
I wrote a rexx tool to load DLLs as permanent into memory or freed
(not palnned to be used when libpathstrict is used.

This help resolving several High shared storage issue where some programs loads the DLL only for the process but under OS2, the free storage isn't reused and left as unusable. This could en by a short of storage into High shared memory (freeze, hung...)

I now set all ffmpeg DLLs to be high loaded using highmem or ABOVE512 and run my rexx tool. All works fine and no more any High shr storage "lost"

My tools has following rules:

ffmpegHigh [option] ddlname or inputfile extraparm

option (required) :
-c   load dlls
-u   free dlls

dllnam  (for a specifuc dll load
or
dllinput file with a list of dlls like the result out of DIR *.DLL /B-D

Extraparm: 1  enable more rc details (can be use in any case)

Note: As read, I can provide an external dll input file which supercedes tool included list.

e.g.
>> Module "X264150" successuffly loaded
>> Module "LIBVPX5" successuffly loaded
>> Module "SDL2" successuffly loaded
>> Module "POSTPR55" successuffly loaded
>> Module "AVDEVI58" successuffly loaded
>> Module "SWRESA3" successuffly loaded
>> Module "SWSCAL5" successuffly loaded
>> Module "AVFILT7" successuffly loaded
>> Module "AVCODE58" successuffly loaded
>> Module "AVFORM58" successuffly loaded
>> Module "AVUTIL56" successuffly loaded

Load of 11 DLLs successfull

 ************

Saved Handle file found and 12 lines read
>> module with handle x"2F200000" successfully freed (rc:0 - X264150)
>> module with handle x"DC1F0000" successfully freed (rc:0 - LIBVPX5)
>> module with handle x"9F1C0000" successfully freed (rc:0 - SDL2)
>> module with handle x"E91A0000" successfully freed (rc:0 - POSTPR55)
>> module with handle x"F01F0000" successfully freed (rc:0 - AVDEVI58)
>> module with handle x"F71F0000" successfully freed (rc:0 - SWRESA3)
>> module with handle x"E31F0000" successfully freed (rc:0 - SWSCAL5)
>> module with handle x"5C200000" successfully freed (rc:0 - AVFILT7)
>> module with handle x"60200000" successfully freed (rc:0 - AVCODE58)
>> module with handle x"E91F0000" successfully freed (rc:0 - AVFORM58)
>> module with handle x"CA1C0000" successfully freed (rc:0 - AVUTIL56)

Free of 11 DLLs successfull


Regards
Remy






 
Title: Re: Shared storage
Post by: guzzi on April 29, 2020, 12:35:05 am
Hi Remy,

I might make sense to use the same parameters as the mozturbo programs, i.e. -l to load dll's instead of -c

Usage: q:\var\temp\FC#37.0\ffturbo.exe [-h] [-l | -u] [-p path]
       -h display this help
       -l load modules
       -u unload modules
       -t turbo function, implies load, uses more memory
       -p specify fully qualified path to directory where the DLLs are located
       -v output version
Title: Re: Shared storage
Post by: Remy on April 29, 2020, 03:27:37 pm
Hi Remy,

I might make sense to use the same parameters as the mozturbo programs, i.e. -l to load dll's instead of -c

Usage: q:\var\temp\FC#37.0\ffturbo.exe [-h] [-l | -u] [-p path]
       -h display this help
       -l load modules
       -u unload modules
       -t turbo function, implies load, uses more memory
       -p specify fully qualified path to directory where the DLLs are located
       -v output version

 ;)

ffmpeg.cmd [-l | -u | -q] [dllname | dlls inputfile] [extra_parm]
Now, available options (one required):
-l  load
-u  free (Unload)
-q  query

dllname or dlls_inputfile are optional parameters (if not ffmpeg dlls have to be loaded..) 

extra parameter after cmd string "1" for detailed error
-h  not added, it is displaied by default if input option isn't valid.

Good suggestion

*Warning*
It can not be used in place of ffturbo (ffturbo takes care of libpathstrict while ffmpegHigh no) 
 http://remydodin.levillage.org/fr/realisations.php?item=6000&id=realisations
Title: Re: Shared storage
Post by: Dariusz Piatkowski on April 29, 2020, 04:33:25 pm
Hi Remy,

Thank you for publishing this. I am thinking this is a benefit each time one runs ffmpeg, that would mean through mplayer, VLC or SMPlay as well.

...
*Warning*
It can not be used in place of ffturbo (ffturbo takes care of libpathstrict while ffmpegHigh no) 
 http://remydodin.levillage.org/fr/realisations.php?item=6000&id=realisations

I do have a follow-up question though. As opposed to ffturbo, it would appear that your setup is persistent only as long as the CMD session remains active, yes?

Code: [Select]
*WARNING* Do not close this "prompt window dialog" to maintain DLL(s) loaded
          And gain benefits of permanent loaded DLL(s) used by applications
          >> Closing this window will free loaded DLL(s) through this tool <<

Therefore, I'm thinking that the script should be run in DETACHed mode during system boot-up?
Title: Re: Shared storage
Post by: Remy on April 29, 2020, 05:16:36 pm
Hi Remy,

Thank you for publishing this. I am thinking this is a benefit each time one runs ffmpeg, that would mean through mplayer, VLC or SMPlay as well.

...
*Warning*
It can not be used in place of ffturbo (ffturbo takes care of libpathstrict while ffmpegHigh no) 
 http://remydodin.levillage.org/fr/realisations.php?item=6000&id=realisations

I do have a follow-up question though. As opposed to ffturbo, it would appear that your setup is persistent only as long as the CMD session remains active, yes?

Code: [Select]
*WARNING* Do not close this "prompt window dialog" to maintain DLL(s) loaded
          And gain benefits of permanent loaded DLL(s) used by applications
          >> Closing this window will free loaded DLL(s) through this tool <<

Therefore, I'm thinking that the script should be run in DETACHed mode during system boot-up?

* It has a sense to use it only if you set ffmpeg dll to load high using above512 or highmem *
dll loaded below 512MB have no storage issue.

First question: Yes
Second question: For those using ffmpeg (with dll flagged to load high) each time after boot, it could be helpful.

Upate notte:. For VLC; VLC.DLL and VLCCORE.DLL with the flag set to load high, have the same issue and can be preloaded too
=> You then create a single file name mydll.dlh (including all ffmpeg dlls as well vlc dlls) and give it as input to ffmpagHgh
ffmpegHigh -l mydll.dlh    (specifying a external file disables included DLL list. You have then to add all DLLs into your external input file)

e.g.
X264150.DLL
LIBVPX5.DLL
SDL2.DLL
POSTPR55.DLL
AVDEVI58.DLL
SWRESA3.DLL
SWSCAL5.DLL
AVFILT7.DLL
AVCODE58.DLL
AVFORM58.DLL
AVUTIL56.DLL
VLCCORE.DLL
VLC.DLL
       
This way, you'll load all DLL's in one command  :D   but of course, you can also run ffmpegHigh twice
ffmpegHigh -l      (will load all ffmpeg dlls)
ffmpegHigh -l vlcdlls.dlh     (external file with only LVC dlls list)

Regards
Rémy   
Title: Re: Shared storage
Post by: Remy on April 30, 2020, 02:16:27 am
Previous post updated
Title: Re: Shared storage
Post by: Remy on April 30, 2020, 10:24:50 am
Hi Dariusz,

I updated ffmpegHigh and added an ffmpegH.exe build which can be started from startup folder using the right parameters.
It start minimized when started with parameters. If no parameters are provided, you'll get a dialog with buttons
web link updated  to download new zip package.

Regards
Remy
Title: Re: Shared storage
Post by: Dariusz Piatkowski on May 02, 2020, 03:57:28 am
Hi Remy!

...I updated ffmpegHigh and added an ffmpegH.exe build which can be started from startup folder using the right parameters...

That's awesome, actually your ffmpeg focused utility is in a sense becoming what I was thinking we were starting to need, which is a system-wide utility to allow you to load any desired DLL high.

If I could whisper a little wish to Santa, here is what I would say ( ;)):

1) give me the ability to have multiple LOAD_HIGH_PROFILES defined
- this way for each app that has DLLs I wish to load high I could point to a destination directory where the DLL exists as well as define a load_high_list file
- the load_high utility would then accept something like the PROFILE_NAME, which would tell it either which load_high_list file to read, or what are the files to be loaded directly by reading from the profile config file

2) if the whole PROFILE thing is too complex then enable the specification of a full pathname to the DLL in the load_high_list file, that way I do not have to make sure the DLLs are in LIBPATH, nor do I have to move them all to a single common subdirectory somewhere on the system

Also, I would ask Santa to troubleshoot this error message I'm currently getting:

Code: [Select]
SAY "    74 +++     single_dll ="     74 +++     single_dll
REX0036: Error 36 running G:\UTIL\MISC\DLL_LH.EXE, line 74: Unmatched "(" in expression

This is the result of the following CLI => 'dll_lh -l dll_lh_list.dlh',
where:
'dll_lh' is the renamed ffmpegh.exe
'dll_lh_list.dlh' is the DLL list file with the following contents:

Code: [Select]
X264150.DLL
LIBVPX5.DLL
SDL2.DLL
POSTPR55.DLL
AVDEVI58.DLL
SWRESA3.DLL
SWSCAL5.DLL
AVFILT7.DLL
AVCODE58.DLL
AVFORM58.DLL
AVUTIL56.DLL
VLCCORE.DLL
VLC.DLL
SNAPWRAP.DLL

All DLLs are present in the execution directory...
Title: Re: Shared storage
Post by: Remy on May 02, 2020, 12:23:52 pm
Hi Remy!

...I updated ffmpegHigh and added an ffmpegH.exe build which can be started from startup folder using the right parameters...

That's awesome, actually your ffmpeg focused utility is in a sense becoming what I was thinking we were starting to need, which is a system-wide utility to allow you to load any desired DLL high.

If I could whisper a little wish to Santa, here is what I would say ( ;)):


window very minimalist
Entry field to specify one dll
File button to import a private file with own dll list  (the included dll list for ffmpeg is than not loaded)
Load button
 




1) give me the ability to have multiple LOAD_HIGH_PROFILES defined
- this way for each app that has DLLs I wish to load high I could point to a destination directory where the DLL exists as well as define a load_high_list file
- the load_high utility would then accept something like the PROFILE_NAME, which would tell it either which load_high_list file to read, or what are the files to be loaded directly by reading from the profile config file

2) if the whole PROFILE thing is too complex then enable the specification of a full pathname to the DLL in the load_high_list file, that way I do not have to make sure the DLLs are in LIBPATH, nor do I have to move them all to a single common subdirectory somewhere on the system

Also, I would ask Santa to troubleshoot this error message I'm currently getting:

Code: [Select]
SAY "    74 +++     single_dll ="     74 +++     single_dll
REX0036: Error 36 running G:\UTIL\MISC\DLL_LH.EXE, line 74: Unmatched "(" in expression

This is the result of the following CLI => 'dll_lh -l dll_lh_list.dlh',
where:
'dll_lh' is the renamed ffmpegh.exe
'dll_lh_list.dlh' is the DLL list file with the following contents:

Code: [Select]
X264150.DLL
LIBVPX5.DLL
SDL2.DLL
POSTPR55.DLL
AVDEVI58.DLL
SWRESA3.DLL
SWSCAL5.DLL
AVFILT7.DLL
AVCODE58.DLL
AVFORM58.DLL
AVUTIL56.DLL
VLCCORE.DLL
VLC.DLL
SNAPWRAP.DLL

All DLLs are present in the execution directory...
"
Hi Dariusz,

About the error, strange, it looks like I forgot or missed a "(".
Try again with the new update, I changed several part of the code to allow loading dlls not into a libpath.

Works like before (hope better :D) and added:
- you can now specify a full path for the DLL in the single dll input field  (do not put the path between quotes)
- The external input file can have a path indication on first line and taking previous lits, you can have now:

rules: the first line is identified as been a path through the "  at beginning and end of path name.
If there aren't present, the entry is handled like a dll entry.

"X:\USR\LOCAL\LIB"
X264150.DLL
LIBVPX5.DLL
SDL2.DLL
POSTPR55.DLL
AVDEVI58.DLL
SWRESA3.DLL
SWSCAL5.DLL
AVFILT7.DLL
AVCODE58.DLL
AVFORM58.DLL
AVUTIL56.DLL
VLCCORE.DLL
VLC.DLL
SNAPWRAP.DLL

Regards,
Remy

A small screen capture (window is minimalist)
Entry field to specify one dll at once                                          (only this dll will be loaded)
File button to select your own file with included dll list          (only dlls listed int file will be loaded)
Load button
Free button
Query button
rc detail checkbox    (a few more information about return code returned)
a  radio just to indicate with 'on' if a load was done
.
.
PS: Not all DLLs, not those loaded through libpathstric as I wrote it  (ex: firefox etc..)

 

Title: Re: Shared storage
Post by: Dariusz Piatkowski on May 02, 2020, 08:01:19 pm
Hi Remy,

Alright sir, so the latest enhancements indeed work great here. I have multiple DLL_LH lists: ffmpeg & vlc. Loading them through the GUI screen works great, however, from CLI it is still the same error.

Is there a chance that RXU.DLL is a problem?

Here is what I have:

6-08-18  8:33a       115,544    124 a---  rxu.dll

Installed as part of a RPM package: rxu-1.a.0-3.oc00.pentium4

Anything else I can try here to debug further?

Also, perhaps a small correction to the README:

Code: [Select]
...
ffmpeghigh (option) dll_name
option:
-c  load dll(s) (it creates a file under path execution named handle.txt required to free loaded dll by rexx)
...

Here when I successfully load the DLLs from the GUI I do not find 'handle.txt' anywhere on my boot drive, nothing in the RAM drive (which is my TMP drive).

I do however find '~ffmpegH' file present in the directory where the DLLs were loaded from, which of course shows the list of the DLLs loaded. Therefore, I think your reference in the README to 'handle.txt' should actually be '~ffmpegH'?

Actually, this last point raises another issue. If I have multiple DLL lists in the same directory, is it not going to be a conflict to load them separately? After all, the DLL load file (~ffempgH) is a unique file and will be overwritten by the 2nd load of the DLLs, right?

Could we perhaps use the name of the input DLL list file instead of a single name like ~ffmpegH?
Title: Re: Shared storage
Post by: Remy on May 02, 2020, 09:28:51 pm
Hi Remy,

Alright sir, so the latest enhancements indeed work great here. I have multiple DLL_LH lists: ffmpeg & vlc. Loading them through the GUI screen works great, however, from CLI it is still the same error.

Is there a chance that RXU.DLL is a problem?

Here is what I have:

6-08-18  8:33a       115,544    124 a---  rxu.dll

Installed as part of a RPM package: rxu-1.a.0-3.oc00.pentium4

Anything else I can try here to debug further?

Also, perhaps a small correction to the README:

Code: [Select]
...
ffmpeghigh (option) dll_name
option:
-c  load dll(s) (it creates a file under path execution named handle.txt required to free loaded dll by rexx)
...

Here when I successfully load the DLLs from the GUI I do not find 'handle.txt' anywhere on my boot drive, nothing in the RAM drive (which is my TMP drive).

I do however find '~ffmpegH' file present in the directory where the DLLs were loaded from, which of course shows the list of the DLLs loaded. Therefore, I think your reference in the README to 'handle.txt' should actually be '~ffmpegH'?

Actually, this last point raises another issue. If I have multiple DLL lists in the same directory, is it not going to be a conflict to load them separately? After all, the DLL load file (~ffempgH) is a unique file and will be overwritten by the 2nd load of the DLLs, right?

Could we perhaps use the name of the input DLL list file instead of a single name like ~ffmpegH?

Hi,

I focus to make the ffmegh.exe best working because this one can be used from startup folder.
The problem is not within rxu, I'm pretty sure it is in my code due there is a litle gap between the exe and the cmd. I'll review the ffmpegfhigh.cmd and update it
Yes, you're right, some information in it a back level compared to ffmpegh.exe and will be part of the update to do.
~ffmpegH replace handle.txt  (this file is more to make things a little bit faster and used for comparisons.  If this file is missing, I query the storage to get the handle and then issue the free (you can do a test, delete the file et do a free). THis is why have it overwritten by new different load is not a problem due I prefer have latest loaded dll's info into this file.

Updated: I found, a missing ")" in my code !

As I wrote, ffmegh is more easy to use and startable from startup folder.
I was very tired during last days with many "white nights" and my vision dropped a little.

I also work on my AVxCAT with many news... to come, I hop this month or next month it all work as expected !     
- added PCM 24bits audio 
- added dynamic gap fc(time) option (audio & video ...)
- use of disc cover.jpg or trackname.jpg or album named image for .....   
- Input file can be Icecast for .... 
- changed concat function
- added audio mix option (with the possibility to change volume level on first audio track  (would be nice to have a background sound while commenting...) 
- lot of changes into auto id3 values (most for Icecast)
- have to end some parts under video
- OS2 mjpeg..
etc...
And more...
etc... 
And corrections (link a little blink at open time due to a window redraw
etc...


To day, some limits appears due to missing TLS/SSL support under ffmpeg   

Due to all changes I do, I had to run many time ffmpeg with several reboot making me searching why and write a quick code to load module et prevent reboot. Now, the only consuming storage is WPS (a WPS restart give back some storage but not all. Of course, no more reboot since I load module permanent (thanks to Dave pointing me to this)

Hope to provide quickly good new and update on the ffmpeghigh.cmd

Best regards
Remy   
Title: Re: Shared storage
Post by: Remy on May 02, 2020, 11:49:46 pm
I spent the time since first update to upgrade and correct latest finding in the code.
Now, both are at the same upgraded code level.
I hope it is better now.

Available on my web page.
Regards
Rémy 
Title: Re: Shared storage
Post by: Dariusz Piatkowski on May 03, 2020, 05:13:59 pm
Hi Remy,

Alright, so the latest version LOAD is working correctly. In other words, I have two separate DLL load lists: FFMPEG and VLC, they can both be loaded from their "source" directory, this is also where the marker file (~ffempgh) is created.

So that's all good.

However, here is what I noticed: if I attempt to un-load the DLLs through the GUI screen I am presented with the following message:

Code: [Select]
     Running ffmpeg (and more) permanent DDLs load into memory   
   ( note1: load / free works for below or above 512MB line     )
   ( note2: DLLs loaded into HIGH SHR is permanent until "free" )
 
   Processing... with options : -u  1
 
 
   !Information! no ~ffmpegH file found !
   (this could be normal at first load request)
   ---------------------------------------------------------
   *WARNING* No "handle" (rexx name) found to proceed, free aborted
 
   Free of 0 DLLs successfull

Subsequently no DLLs are unloaded. Now in the bigger scheme of things this is not a problem per say because closing the program does successfully un-load the DLLs, however, this functionality was working in previous versions. So I'm guessing something sneaked through here?
Title: Re: Shared storage
Post by: Remy on May 04, 2020, 01:44:25 am
Hi Remy,

Alright, so the latest version LOAD is working correctly. In other words, I have two separate DLL load lists: FFMPEG and VLC, they can both be loaded from their "source" directory, this is also where the marker file (~ffempgh) is created.

So that's all good.

However, here is what I noticed: if I attempt to un-load the DLLs through the GUI screen I am presented with the following message:

Code: [Select]
     Running ffmpeg (and more) permanent DDLs load into memory   
   ( note1: load / free works for below or above 512MB line     )
   ( note2: DLLs loaded into HIGH SHR is permanent until "free" )
 
   Processing... with options : -u  1
 
 
   !Information! no ~ffmpegH file found !
   (this could be normal at first load request)
   ---------------------------------------------------------
   *WARNING* No "handle" (rexx name) found to proceed, free aborted
 
   Free of 0 DLLs successfull

Subsequently no DLLs are unloaded. Now in the bigger scheme of things this is not a problem per say because closing the program does successfully un-load the DLLs, however, this functionality was working in previous versions. So I'm guessing something sneaked through here?

Well normal  ::)
(some code missing)

I upated both
online
Title: Re: Shared storage
Post by: Dariusz Piatkowski on May 10, 2020, 10:14:14 pm
Hi Remy,

Alright, so I now have a total of three separate DLL LH lists:

1) ffmpeg
2) vlc
3) usr_dll

In particular, for the moment I'm loading just the usr_dll and that has a single DLL in it: SNAPWRAP.DLL, which several other apps use.

OK, so that load seems to happen fine, but I am getting something strange in terms of the outputs of both the 'Query' and 'Free' commands.

1) QUERY (with the RC option turned ON)
Code: [Select]
Running ffmpeg (and more) permanent DDLs load into memory   
   ( note1: load / free works for below or above 512MB line     )
   ( note2: DLLs loaded into HIGH SHR is permanent until "free" )
 
   Processing... with options : -q  1
 
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for X264150 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for LIBVPX5 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for SDL2 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for POSTPR55 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVDEVI58 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for SWRESA3 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for SWSCAL5 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVFILT7 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVCODE58 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVFORM58 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVUTIL56 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for VLCCORE [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for VLC [126]
 
   DLL(s) query handle done
 
   note 1: Found = at least, modules where loaded one time post boot
   note 2: Found = Module may be loaded or not (freed)

2) FREE (with the RC optioned turned ON)
Code: [Select]

     Running ffmpeg (and more) permanent DDLs load into memory   
   ( note1: load / free works for below or above 512MB line     )
   ( note2: DLLs loaded into HIGH SHR is permanent until "free" )
 
   Processing... with options : -u  1
 
 
   !Information! no ~ffmpegH file found !
   (this could be normal at first load request)
   ---------------------------------------------------------
   No query handle found for DLL: X264150 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: LIBVPX5 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: SDL2 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: POSTPR55 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVDEVI58 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: SWRESA3 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: SWSCAL5 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVFILT7 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVCODE58 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVFORM58 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVUTIL56 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: VLCCORE -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: VLC -  an RC99 will be set fo this DLL on free
   End of dynamic handle acquisition (query), process continues...
 
   -  Error during free of module handle x"        " (rc:99 - X264150)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - LIBVPX5)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - SDL2)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - POSTPR55)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVDEVI58)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - SWRESA3)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - SWSCAL5)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVFILT7)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVCODE58)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVFORM58)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVUTIL56)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - VLCCORE)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - VLC)
   -  detail: [ERROR_NO_HANDLE_FOUND]
 
   Successfully freed  DLLs : 0, expected : 13
   Not freed count : 13
 

...but strangely enough SNAPWRAP.DLL is nowhere to be found...???
Title: Re: Shared storage
Post by: Dave Yeo on May 10, 2020, 10:54:56 pm
I hope you aren't marking snapwrap.dll to load high, remember that anything that accesses a 16 bit function should not be loaded high. Not sure about snapwrap but better safe. Same with SDL and SDL2, I've never been brave enough to mark them to load high as so much of the multimedia stuff is 16 bit.
Title: Re: Shared storage
Post by: Dariusz Piatkowski on May 10, 2020, 11:10:26 pm
Hi Dave,

I hope you aren't marking snapwrap.dll to load high, remember that anything that accesses a 16 bit function should not be loaded high...

Actually...yup, loading high and both VLC and mplayer certainly seems to be fine with it. No issues so far anyways.

Not sure that this particular DLL has ever caused a problem in the past, however in my attempts to try to minimize the amount of shared-memory consumption I figured I would toss as many things into high-memory as possible and see how they pan out.

I admit, it is a bit of a brute force approach at best!
Title: Re: Shared storage
Post by: Remy on May 10, 2020, 11:13:43 pm
Hi Remy,

Alright, so I now have a total of three separate DLL LH lists:

1) ffmpeg
2) vlc
3) usr_dll

In particular, for the moment I'm loading just the usr_dll and that has a single DLL in it: SNAPWRAP.DLL, which several other apps use.

OK, so that load seems to happen fine, but I am getting something strange in terms of the outputs of both the 'Query' and 'Free' commands.

1) QUERY (with the RC option turned ON)
Code: [Select]
Running ffmpeg (and more) permanent DDLs load into memory   
   ( note1: load / free works for below or above 512MB line     )
   ( note2: DLLs loaded into HIGH SHR is permanent until "free" )
 
   Processing... with options : -q  1
 
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for X264150 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for LIBVPX5 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for SDL2 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for POSTPR55 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVDEVI58 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for SWRESA3 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for SWSCAL5 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVFILT7 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVCODE58 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVFORM58 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for AVUTIL56 [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for VLCCORE [126]
   -  detail: [ERROR_CODE_126_NOT_FOUND]
   No handle for VLC [126]
 
   DLL(s) query handle done
 
   note 1: Found = at least, modules where loaded one time post boot
   note 2: Found = Module may be loaded or not (freed)

2) FREE (with the RC optioned turned ON)
Code: [Select]

     Running ffmpeg (and more) permanent DDLs load into memory   
   ( note1: load / free works for below or above 512MB line     )
   ( note2: DLLs loaded into HIGH SHR is permanent until "free" )
 
   Processing... with options : -u  1
 
 
   !Information! no ~ffmpegH file found !
   (this could be normal at first load request)
   ---------------------------------------------------------
   No query handle found for DLL: X264150 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: LIBVPX5 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: SDL2 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: POSTPR55 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVDEVI58 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: SWRESA3 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: SWSCAL5 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVFILT7 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVCODE58 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVFORM58 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: AVUTIL56 -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: VLCCORE -  an RC99 will be set fo this DLL on free
   No query handle found for DLL: VLC -  an RC99 will be set fo this DLL on free
   End of dynamic handle acquisition (query), process continues...
 
   -  Error during free of module handle x"        " (rc:99 - X264150)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - LIBVPX5)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - SDL2)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - POSTPR55)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVDEVI58)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - SWRESA3)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - SWSCAL5)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVFILT7)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVCODE58)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVFORM58)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - AVUTIL56)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - VLCCORE)
   -  detail: [ERROR_NO_HANDLE_FOUND]
   -  Error during free of module handle x"        " (rc:99 - VLC)
   -  detail: [ERROR_NO_HANDLE_FOUND]
 
   Successfully freed  DLLs : 0, expected : 13
   Not freed count : 13
 

...but strangely enough SNAPWRAP.DLL is nowhere to be found...???

Hi Dariusz,

Did you load ffmpeg dlls prior the query ?
If you would like check the loaded DLL, be sure to specify the dll list for the -q and -u option.
From what I could see, rxu function was unable to find handle for the given dlls.

Note: If you loaded modules and closed the prompt window under which you run the cmd file, DLLs are then freed...
           (the prompt window must be left open or use the exe build and do not close it)


Regards
Rémy

I would suggest to do as Dave wrote.   


ps: 1/ a remaining problem seems to be the WPS  (WPS restart helps to free some excessive storage it uses).
       2/ My system is running since I have my DLLs from ffmpeg and vlc loaded high 'including SDL2) and it seems stable despite I run a lot of video and audio files from different type. mp3, ogg, wav etc... (As soon I'll have a problem, I'll remove SDL2 from my list)         




 
Title: Re: Shared storage
Post by: Dariusz Piatkowski on May 10, 2020, 11:25:03 pm
Remy,

...Did you load ffmpeg dlls prior the query ?
If you would like check the loaded DLL, be sure to specify the dll list for the -q and -u option.
From what I could see, rxu function was unable to find handle for the given dlls...

No, I did not load any other DLLs high, just SNAPWRAP.DLL.

I am not sure why the other DLLs show up in both the Free and Query lists, I assume that it's because your code is looking for the presence of the '~ffmpegH' file and getting that info from there?

On my system today I have the following files present:

Code: [Select]
Directory of G:\

 5-07-20 11:32a           483      0 a---  ~ffmpegH
        1 file(s)         483 bytes used

Directory of G:\apps\multimedia\vlc2\usr\local\lib

 5-07-20  5:34p           179      0 a---  ~ffmpegH
        1 file(s)         179 bytes used

Directory of G:\usr\dll

 5-09-20 12:42p            57      0 a---  ~ffmpegH
        1 file(s)          57 bytes used

They are each found in the 'source DLL directory' of where the maching *.dlh load file points.

The following are the contents in each file:

1)
Code: [Select]
[G:\]type ~ffmpegH
DLLhnd.0='11'
DLLhnd.1='250D0000''Dll_list.1='X264150'
DLLhnd.2='4B0D0000''Dll_list.2='LIBVPX5'
DLLhnd.3='810D0000''Dll_list.3='SDL2'
DLLhnd.4='820A0000''Dll_list.4='POSTPR55'
DLLhnd.5='2D0D0000''Dll_list.5='AVDEVI58'
DLLhnd.6='A50A0000''Dll_list.6='SWRESA3'
DLLhnd.7='870A0000''Dll_list.7='SWSCAL5'
DLLhnd.8='900A0000''Dll_list.8='AVFILT7'
DLLhnd.9='910A0000''Dll_list.9='AVCODE58'
DLLhnd.10='460D0000''Dll_list.10='AVFORM58'
DLLhnd.11='8B0A0000''Dll_list.11='AVUTIL56'

2)
Code: [Select]
[G:\apps\multimedia\vlc2\usr\local\lib]type ~ffmpegH
DLLhnd.0='4'
DLLhnd.1='4D190000''Dll_list.1='SNAPWRAP'
DLLhnd.2='4B190000''Dll_list.2='VLC'
DLLhnd.3='69190000''Dll_list.3='VLCCORE'
DLLhnd.4='5B190000''Dll_list.4='DVDCSS2'

3)
Code: [Select]
[G:\usr\dll]type ~ffmpegH
DLLhnd.0='1'
DLLhnd.1='51190000''Dll_list.1='SNAPWRAP'

So by the looks of it, it feels like the one I'm actually loading (usr_dll) which only has SNAPWRAP.DLL gets lost and is not read in anymore?
Title: Re: Shared storage
Post by: Remy on May 11, 2020, 12:02:46 am
Hum !

Did you run ffmpeghigh -q usr_dll.DLH    to do the query for the corresponding DLLs ?
Wihtout specifying the dll list, no chance to find DLLs loaded out of the default defined into ffmpeghigh

This is the case for -l  -q or -u option. in each case, add the file including a dlls list to get information from these dlls.                       
Title: Re: Shared storage
Post by: Dariusz Piatkowski on May 11, 2020, 12:22:11 am
Ahhh!!!!!

Eureka...I got it!!!!  ;D

...Did you run ffmpeghigh -q usr_dll.DLH    to do the query for the corresponding DLLs ?
Wihtout specifying the dll list, no chance to find DLLs loaded out of the default defined into ffmpeghigh

This is the case for -l  -q or -u option. in each case, add the file including a dlls list to get information from these dlls.                       

No, see the way I was running these is to define the program object where I would pass the '-l dll_list.dlh' parameter. The GUI window then comes up, but the dll/load_list text field is empty. Therefore, each time I would select 'Query' or 'Free' no wonder i was getting the results I saw. I was anticipating / expecting that the program surely knew what DLLs I was inquiring about since I passed that list in at start-up-time as an input parameter after all.

So for example, having started the program in that fashion, when I press the radio button next to the '*DLL' label I see the following output:
Code: [Select]
...   
Processing... with options : -l usr_dll_list.dlh
 
   Dll(s) path : "G:\USR\DLL"
   Path to DLL(s) found, process will continue and try loading DLL(s)
   >> Module "SNAPWRAP" successuffly loaded
 
   Load of 1 DLLs successfull
   (ps:under test mode, wait a few seconds before trying the "free" option)
...

In this case, the program certainly knows what DLL it is supposed to watch for. But, if I used the 'Query' or 'Free' button instead, it shows something different:

Code: [Select]
...   
Processing... with options : -q 
 
 
   DLL(s) query handle done
...

This behaviour is what confused me. Because in the 1st example above the program knows what DLL it is looking at (that which was loaded from the INPUT command line) but in the 2nd case it does not, that is because when pressing the 'Free' or 'Query' button it's actually attempting to take the input from the text-feld instead.

Now that I understand how the GUI part works the results are correct whenever I pass the actual dll_list.dlh I originally started the program with.

So here is a request/suggestion: when such a DLL list is passed along to the program at load time, can you automatically list that name in the text field? That way when you have multiple GUIs started you do not have to wonder which one is which...and best of all, even if you have a single one running you do not have to re-type the dll_list.dlh each time into that field.
Title: Re: Shared storage
Post by: Remy on May 11, 2020, 12:55:15 am

So here is a request/suggestion: when such a DLL list is passed along to the program at load time, can you automatically list that name in the text field? That way when you have multiple GUIs started you do not have to wonder which one is which...and best of all, even if you have a single one running you do not have to re-type the dll_list.dlh each time into that field.

Ok, done.

Just get the new ffmpeghigh.zip file

note: added a help button and when you click(s) on the  radio button (used to show load was done or not), you can show the message panel with the load result.
Title: Re: Shared storage
Post by: Dariusz Piatkowski on May 11, 2020, 02:50:41 pm
Hi Remy,

Ok, done.

Just get the new ffmpeghigh.zip file

note: added a help button and when you click(s) on the  radio button (used to show load was done or not), you can show the message panel with the load result.

Alright, so the input param is now preserved, works great.

However, I did run into a follow-up issue: attempting to select the 'Query' button (or any other one for that matter) for the 2nd time produces the following error in DrRexx console:

Code: [Select]
SAY "   501 +++                   dllfile ="    501 +++                   dllfile
REX0026: Error 26 running G:\UTIL\MISC\DLL_LH.EXE, line 501: Invalid whole number
   208 +++                 Call read_dll_file;
    29 +++   Call ffmpegHigh;

It's almost like the program loses track of the input filename that was passed to it??? (just a guess based on what the above error looks like).
Title: Re: Shared storage
Post by: Remy on May 11, 2020, 06:53:20 pm
Hi Remy,

Ok, done.

Just get the new ffmpeghigh.zip file

note: added a help button and when you click(s) on the  radio button (used to show load was done or not), you can show the message panel with the load result.

Alright, so the input param is now preserved, works great.

However, I did run into a follow-up issue: attempting to select the 'Query' button (or any other one for that matter) for the 2nd time produces the following error in DrRexx console:

Code: [Select]
SAY "   501 +++                   dllfile ="    501 +++                   dllfile
REX0026: Error 26 running G:\UTIL\MISC\DLL_LH.EXE, line 501: Invalid whole number
   208 +++                 Call read_dll_file;
    29 +++   Call ffmpegHigh;

It's almost like the program loses track of the input filename that was passed to it??? (just a guess based on what the above error looks like).

Strange !
I   see where it fails but I could not reproduce nor do I see why you've got the invalid whole number
I added a few verification to check not going into an invalid whole number into this part of the code as well. Also added routine if the input file could not be read "already open" or other errors ( if ffmpegh started well with parameters but the radio button isn't on ! them, click on it to display latest messages)

Here it is for your test (only replace the impacted file from the package by this one)
   
Hope it works better.
Regards
Rémy
Title: Re: Shared storage
Post by: Dariusz Piatkowski on May 11, 2020, 07:51:51 pm
Remy!

I think I've got it...

..I   see where it fails but I could not reproduce nor do I see why you've got the invalid whole number
I added a few verification to check not going into an invalid whole number into this part of the code as well. Also added routine if the input file could not be read "already open" or other errors ( if ffmpegh started well with parameters but the radio button isn't on ! them, click on it to display latest messages)...

Looks to me like the program is not using a fully qualified DLL_LIST filename due to no fault of it's own. It seems this is the result of the WPS program object defintion I created, where I have the following:

PATH AND FILENAME = G:\UTIL\MISC\DLL_LH.EXE
INPUT_PARAMETERS = -c usr_dll_list.dlh
WORKING_DIRECTORY = (blank)

Here is why I say that:

1) start the program with the input command line as follows '

The intial load goes fine, as does the initial invocating of something like 'Query', the results are:

Code: [Select]
...
   Processing... with options : -q usr_dll_list.dlh 1
 
   Dll(s) path : "G:\USR\DLL"
   Path to DLL(s) found, process will continue and try loading DLL(s)
   *Information* found module with handle x"3D210000"
    [G:\USR\DLL\SNAPWRAP.DLL]
 
   DLL(s) query handle done
...

...but afterwards any attempts to use 'Query' or 'Free' all result in the following error output:
 
Code: [Select]
...
    ** ERROR ** an error occured reading input file including dll list
    Verify input file for possible cause :
    - File aleady open
    - File size = 0
    - First line has an incorrect path syntax
       (path must be specified between double quotes e.g. "C:\usr\lib")
    - Other possible issue
 
    >>>>  current request aborted
...

NOTE: whatever way you're handling the trap is also preventing (here on my machine) me from going back to the GUI window itself to actually close the program, instead I use 'kill' which does successfully close it.

2) re-do the start, same input command line, but now I actually modify the text-field which stores the input parameter the program was originally started with, so instead of just using the filename (usr_dll_list.dlh) I now use a fully qualified name so that includes the complete path (g:\util\misc\usr_dll_list.dlh), this produces the following successfull output:

Code: [Select]
   Processing... with options : -q g:\util\misc\usr_dll_list.dlh 1
 
   Dll(s) path : "G:\USR\DLL"
   Path to DLL(s) found, process will continue and try loading DLL(s)
   *Information* found module with handle x"F71D0000"
    [G:\USR\DLL\SNAPWRAP.DLL]
 
   DLL(s) query handle done

The way to get around this is to modify the WPS Program object to set the INPUT_PARAMETER to show a fully qualified path, which I tested and am happy to report that it works.

I did not try setting the WORKING_DIRECTORY in the Program object to see what impact, if any, it had.

Alright, I would say this works. Unless there were any other changes in this test build (other than the DEBUG stuff) I think I can go back to using the previous release and just use a fully qualified pathname as the input parameter.

Thanks Remy!!!
Title: Re: Shared storage
Post by: Remy on May 12, 2020, 01:59:58 am
Hi Dariusz,

With your last update, I found what's going wrong and why you have to specify the path under WPS pgm icon.
Now, I corrected it and I display from under which path ~ffmpeh is taken (same as dll to load) if exist.
I added a few more verification and check not disabling the main dialog :D (what you had after the new error messages)
I added a quick [e] button to edit the input file with DLLs in case some changes are needed.

Please take the new build I just put online on my site (avxcat page)

Regards
Rémy