Shared storage

[Remy]

Hi !

Any idea why an older ffmpeg compiled with gcc 4.4.4 doesn't left any shared storage holes at close time while ffmpeg compiled with gcc 5.5.0 left about 20MB (all dlls size) of free unusable holes in the shared memory at close time ?

(FFmpeg or ffprobe)

Updt: problem occures as well with ffplay but in all cases only with dll's loaded above 512MB line  (when below, no issue)


Regards
Rémy

[Remy]

Well, same raison as for firefox for which ffturbo was build.

I tested some rexx functions which seems to well work.
Working with this now.

Cheers/2

[Remy]

Problem resolved and it works pretty well.

I coded a small rex which loads ffmpeg dlls (set to be high loaded by ABOVE512 or highmem) permanently into high shared storage. (all ffmpeg 4.0.2 eligible dlls are coded in the rexx. It work like ffturbo (without the need to have a running background process) but ffturbo is more complex due to libpathstrict...

If you are interested by my rexx code, send me an email from under my web site, same as AVxCAT, CLAMAVGUI...
remydodin*

[Remy]

I wrote a rexx tool to load DLLs as permanent into memory or freed
(not palnned to be used when libpathstrict is used.

This help resolving several High shared storage issue where some programs loads the DLL only for the process but under OS2, the free storage isn't reused and left as unusable. This could en by a short of storage into High shared memory (freeze, hung...)

I now set all ffmpeg DLLs to be high loaded using highmem or ABOVE512 and run my rexx tool. All works fine and no more any High shr storage "lost"

My tools has following rules:

ffmpegHigh [option] ddlname or inputfile extraparm

option (required) :
-c   load dlls
-u   free dlls

dllnam  (for a specifuc dll load
or
dllinput file with a list of dlls like the result out of DIR *.DLL /B-D

Extraparm: 1  enable more rc details (can be use in any case)

Note: As read, I can provide an external dll input file which supercedes tool included list.

e.g.
>> Module "X264150" successuffly loaded
>> Module "LIBVPX5" successuffly loaded
>> Module "SDL2" successuffly loaded
>> Module "POSTPR55" successuffly loaded
>> Module "AVDEVI58" successuffly loaded
>> Module "SWRESA3" successuffly loaded
>> Module "SWSCAL5" successuffly loaded
>> Module "AVFILT7" successuffly loaded
>> Module "AVCODE58" successuffly loaded
>> Module "AVFORM58" successuffly loaded
>> Module "AVUTIL56" successuffly loaded

Load of 11 DLLs successfull

 ************

Saved Handle file found and 12 lines read
>> module with handle x"2F200000" successfully freed (rc:0 - X264150)
>> module with handle x"DC1F0000" successfully freed (rc:0 - LIBVPX5)
>> module with handle x"9F1C0000" successfully freed (rc:0 - SDL2)
>> module with handle x"E91A0000" successfully freed (rc:0 - POSTPR55)
>> module with handle x"F01F0000" successfully freed (rc:0 - AVDEVI58)
>> module with handle x"F71F0000" successfully freed (rc:0 - SWRESA3)
>> module with handle x"E31F0000" successfully freed (rc:0 - SWSCAL5)
>> module with handle x"5C200000" successfully freed (rc:0 - AVFILT7)
>> module with handle x"60200000" successfully freed (rc:0 - AVCODE58)
>> module with handle x"E91F0000" successfully freed (rc:0 - AVFORM58)
>> module with handle x"CA1C0000" successfully freed (rc:0 - AVUTIL56)

Free of 11 DLLs successfull


Regards
Remy






 

[guzzi]

Hi Remy,

I might make sense to use the same parameters as the mozturbo programs, i.e. -l to load dll's instead of -c

Usage: q:\var\temp\FC#37.0\ffturbo.exe [-h] [-l | -u] [-p path]
       -h display this help
       -l load modules
       -u unload modules
       -t turbo function, implies load, uses more memory
       -p specify fully qualified path to directory where the DLLs are located
       -v output version

[Remy]

Hi Remy,

I might make sense to use the same parameters as the mozturbo programs, i.e. -l to load dll's instead of -c

Usage: q:\var\temp\FC#37.0\ffturbo.exe [-h] [-l | -u] [-p path]
       -h display this help
       -l load modules
       -u unload modules
       -t turbo function, implies load, uses more memory
       -p specify fully qualified path to directory where the DLLs are located
       -v output version

 

ffmpeg.cmd [-l | -u | -q] [dllname | dlls inputfile] [extra_parm]
Now, available options (one required):
-l  load
-u  free (Unload)
-q  query

dllname or dlls_inputfile are optional parameters (if not ffmpeg dlls have to be loaded..) 

extra parameter after cmd string "1" for detailed error
-h  not added, it is displaied by default if input option isn't valid.

Good suggestion

*Warning*
It can not be used in place of ffturbo (ffturbo takes care of libpathstrict while ffmpegHigh no) 
 http://remydodin.levillage.org/fr/realisations.php?item=6000&id=realisations

[Dariusz Piatkowski]

Hi Remy,

Thank you for publishing this. I am thinking this is a benefit each time one runs ffmpeg, that would mean through mplayer, VLC or SMPlay as well.

...
*Warning*
It can not be used in place of ffturbo (ffturbo takes care of libpathstrict while ffmpegHigh no) 
 http://remydodin.levillage.org/fr/realisations.php?item=6000&id=realisations

I do have a follow-up question though. As opposed to ffturbo, it would appear that your setup is persistent only as long as the CMD session remains active, yes?

Code: [Select]

*WARNING* Do not close this "prompt window dialog" to maintain DLL(s) loaded
          And gain benefits of permanent loaded DLL(s) used by applications
          >> Closing this window will free loaded DLL(s) through this tool <<

Therefore, I'm thinking that the script should be run in DETACHed mode during system boot-up?

[Remy]

Hi Remy,

Thank you for publishing this. I am thinking this is a benefit each time one runs ffmpeg, that would mean through mplayer, VLC or SMPlay as well.

...
*Warning*
It can not be used in place of ffturbo (ffturbo takes care of libpathstrict while ffmpegHigh no) 
 http://remydodin.levillage.org/fr/realisations.php?item=6000&id=realisations

I do have a follow-up question though. As opposed to ffturbo, it would appear that your setup is persistent only as long as the CMD session remains active, yes?

Code: [Select]

*WARNING* Do not close this "prompt window dialog" to maintain DLL(s) loaded
          And gain benefits of permanent loaded DLL(s) used by applications
          >> Closing this window will free loaded DLL(s) through this tool <<

Therefore, I'm thinking that the script should be run in DETACHed mode during system boot-up?

* It has a sense to use it only if you set ffmpeg dll to load high using above512 or highmem *
dll loaded below 512MB have no storage issue.

First question: Yes
Second question: For those using ffmpeg (with dll flagged to load high) each time after boot, it could be helpful.

Upate notte:. For VLC; VLC.DLL and VLCCORE.DLL with the flag set to load high, have the same issue and can be preloaded too
=> You then create a single file name mydll.dlh (including all ffmpeg dlls as well vlc dlls) and give it as input to ffmpagHgh
ffmpegHigh -l mydll.dlh    (specifying a external file disables included DLL list. You have then to add all DLLs into your external input file)

e.g.
X264150.DLL
LIBVPX5.DLL
SDL2.DLL
POSTPR55.DLL
AVDEVI58.DLL
SWRESA3.DLL
SWSCAL5.DLL
AVFILT7.DLL
AVCODE58.DLL
AVFORM58.DLL
AVUTIL56.DLL
VLCCORE.DLL
VLC.DLL
       
This way, you'll load all DLL's in one command     but of course, you can also run ffmpegHigh twice
ffmpegHigh -l      (will load all ffmpeg dlls)
ffmpegHigh -l vlcdlls.dlh     (external file with only LVC dlls list)

Regards
Rémy   

[Remy]

Previous post updated

[Remy]

Hi Dariusz,

I updated ffmpegHigh and added an ffmpegH.exe build which can be started from startup folder using the right parameters.
It start minimized when started with parameters. If no parameters are provided, you'll get a dialog with buttons
web link updated  to download new zip package.

Regards
Remy

[Dariusz Piatkowski]

Hi Remy!

...I updated ffmpegHigh and added an ffmpegH.exe build which can be started from startup folder using the right parameters...

That's awesome, actually your ffmpeg focused utility is in a sense becoming what I was thinking we were starting to need, which is a system-wide utility to allow you to load any desired DLL high.

If I could whisper a little wish to Santa, here is what I would say ( ):

1) give me the ability to have multiple LOAD_HIGH_PROFILES defined
- this way for each app that has DLLs I wish to load high I could point to a destination directory where the DLL exists as well as define a load_high_list file
- the load_high utility would then accept something like the PROFILE_NAME, which would tell it either which load_high_list file to read, or what are the files to be loaded directly by reading from the profile config file

2) if the whole PROFILE thing is too complex then enable the specification of a full pathname to the DLL in the load_high_list file, that way I do not have to make sure the DLLs are in LIBPATH, nor do I have to move them all to a single common subdirectory somewhere on the system

Also, I would ask Santa to troubleshoot this error message I'm currently getting:

Code: [Select]

SAY "    74 +++     single_dll ="     74 +++     single_dll
REX0036: Error 36 running G:\UTIL\MISC\DLL_LH.EXE, line 74: Unmatched "(" in expression

This is the result of the following CLI => 'dll_lh -l dll_lh_list.dlh',
where:
'dll_lh' is the renamed ffmpegh.exe
'dll_lh_list.dlh' is the DLL list file with the following contents:

Code: [Select]

X264150.DLL
LIBVPX5.DLL
SDL2.DLL
POSTPR55.DLL
AVDEVI58.DLL
SWRESA3.DLL
SWSCAL5.DLL
AVFILT7.DLL
AVCODE58.DLL
AVFORM58.DLL
AVUTIL56.DLL
VLCCORE.DLL
VLC.DLL
SNAPWRAP.DLL

All DLLs are present in the execution directory...

[Remy]

Hi Remy!

...I updated ffmpegHigh and added an ffmpegH.exe build which can be started from startup folder using the right parameters...

That's awesome, actually your ffmpeg focused utility is in a sense becoming what I was thinking we were starting to need, which is a system-wide utility to allow you to load any desired DLL high.

If I could whisper a little wish to Santa, here is what I would say ( ):


window very minimalist
Entry field to specify one dll
File button to import a private file with own dll list  (the included dll list for ffmpeg is than not loaded)
Load button
 




1) give me the ability to have multiple LOAD_HIGH_PROFILES defined
- this way for each app that has DLLs I wish to load high I could point to a destination directory where the DLL exists as well as define a load_high_list file
- the load_high utility would then accept something like the PROFILE_NAME, which would tell it either which load_high_list file to read, or what are the files to be loaded directly by reading from the profile config file

2) if the whole PROFILE thing is too complex then enable the specification of a full pathname to the DLL in the load_high_list file, that way I do not have to make sure the DLLs are in LIBPATH, nor do I have to move them all to a single common subdirectory somewhere on the system

Also, I would ask Santa to troubleshoot this error message I'm currently getting:

Code: [Select]

SAY "    74 +++     single_dll ="     74 +++     single_dll
REX0036: Error 36 running G:\UTIL\MISC\DLL_LH.EXE, line 74: Unmatched "(" in expression

This is the result of the following CLI => 'dll_lh -l dll_lh_list.dlh',
where:
'dll_lh' is the renamed ffmpegh.exe
'dll_lh_list.dlh' is the DLL list file with the following contents:

Code: [Select]

X264150.DLL
LIBVPX5.DLL
SDL2.DLL
POSTPR55.DLL
AVDEVI58.DLL
SWRESA3.DLL
SWSCAL5.DLL
AVFILT7.DLL
AVCODE58.DLL
AVFORM58.DLL
AVUTIL56.DLL
VLCCORE.DLL
VLC.DLL
SNAPWRAP.DLL

All DLLs are present in the execution directory...

"
Hi Dariusz,

About the error, strange, it looks like I forgot or missed a "(".
Try again with the new update, I changed several part of the code to allow loading dlls not into a libpath.

Works like before (hope better ) and added:
- you can now specify a full path for the DLL in the single dll input field  (do not put the path between quotes)
- The external input file can have a path indication on first line and taking previous lits, you can have now:

rules: the first line is identified as been a path through the "  at beginning and end of path name.
If there aren't present, the entry is handled like a dll entry.

"X:\USR\LOCAL\LIB"
X264150.DLL
LIBVPX5.DLL
SDL2.DLL
POSTPR55.DLL
AVDEVI58.DLL
SWRESA3.DLL
SWSCAL5.DLL
AVFILT7.DLL
AVCODE58.DLL
AVFORM58.DLL
AVUTIL56.DLL
VLCCORE.DLL
VLC.DLL
SNAPWRAP.DLL

Regards,
Remy

A small screen capture (window is minimalist)
Entry field to specify one dll at once                                          (only this dll will be loaded)
File button to select your own file with included dll list          (only dlls listed int file will be loaded)
Load button
Free button
Query button
rc detail checkbox    (a few more information about return code returned)
a  radio just to indicate with 'on' if a load was done
.
.
PS: Not all DLLs, not those loaded through libpathstric as I wrote it  (ex: firefox etc..)

 

[Dariusz Piatkowski]

Hi Remy,

Alright sir, so the latest enhancements indeed work great here. I have multiple DLL_LH lists: ffmpeg & vlc. Loading them through the GUI screen works great, however, from CLI it is still the same error.

Is there a chance that RXU.DLL is a problem?

Here is what I have:

6-08-18  8:33a       115,544    124 a---  rxu.dll

Installed as part of a RPM package: rxu-1.a.0-3.oc00.pentium4

Anything else I can try here to debug further?

Also, perhaps a small correction to the README:

Code: [Select]

...
ffmpeghigh (option) dll_name
option:
-c  load dll(s) (it creates a file under path execution named handle.txt required to free loaded dll by rexx)
...

Here when I successfully load the DLLs from the GUI I do not find 'handle.txt' anywhere on my boot drive, nothing in the RAM drive (which is my TMP drive).

I do however find '~ffmpegH' file present in the directory where the DLLs were loaded from, which of course shows the list of the DLLs loaded. Therefore, I think your reference in the README to 'handle.txt' should actually be '~ffmpegH'?

Actually, this last point raises another issue. If I have multiple DLL lists in the same directory, is it not going to be a conflict to load them separately? After all, the DLL load file (~ffempgH) is a unique file and will be overwritten by the 2nd load of the DLLs, right?

Could we perhaps use the name of the input DLL list file instead of a single name like ~ffmpegH?

[Remy]

Hi Remy,

Alright sir, so the latest enhancements indeed work great here. I have multiple DLL_LH lists: ffmpeg & vlc. Loading them through the GUI screen works great, however, from CLI it is still the same error.

Is there a chance that RXU.DLL is a problem?

Here is what I have:

6-08-18  8:33a       115,544    124 a---  rxu.dll

Installed as part of a RPM package: rxu-1.a.0-3.oc00.pentium4

Anything else I can try here to debug further?

Also, perhaps a small correction to the README:

Code: [Select]

...
ffmpeghigh (option) dll_name
option:
-c  load dll(s) (it creates a file under path execution named handle.txt required to free loaded dll by rexx)
...

Here when I successfully load the DLLs from the GUI I do not find 'handle.txt' anywhere on my boot drive, nothing in the RAM drive (which is my TMP drive).

I do however find '~ffmpegH' file present in the directory where the DLLs were loaded from, which of course shows the list of the DLLs loaded. Therefore, I think your reference in the README to 'handle.txt' should actually be '~ffmpegH'?

Actually, this last point raises another issue. If I have multiple DLL lists in the same directory, is it not going to be a conflict to load them separately? After all, the DLL load file (~ffempgH) is a unique file and will be overwritten by the 2nd load of the DLLs, right?

Could we perhaps use the name of the input DLL list file instead of a single name like ~ffmpegH?

Hi,

I focus to make the ffmegh.exe best working because this one can be used from startup folder.
The problem is not within rxu, I'm pretty sure it is in my code due there is a litle gap between the exe and the cmd. I'll review the ffmpegfhigh.cmd and update it
Yes, you're right, some information in it a back level compared to ffmpegh.exe and will be part of the update to do.
~ffmpegH replace handle.txt  (this file is more to make things a little bit faster and used for comparisons.  If this file is missing, I query the storage to get the handle and then issue the free (you can do a test, delete the file et do a free). THis is why have it overwritten by new different load is not a problem due I prefer have latest loaded dll's info into this file.

Updated: I found, a missing ")" in my code !

As I wrote, ffmegh is more easy to use and startable from startup folder.
I was very tired during last days with many "white nights" and my vision dropped a little.

I also work on my AVxCAT with many news... to come, I hop this month or next month it all work as expected !     
- added PCM 24bits audio 
- added dynamic gap fc(time) option (audio & video ...)
- use of disc cover.jpg or trackname.jpg or album named image for .....   
- Input file can be Icecast for .... 
- changed concat function
- added audio mix option (with the possibility to change volume level on first audio track  (would be nice to have a background sound while commenting...) 
- lot of changes into auto id3 values (most for Icecast)
- have to end some parts under video
- OS2 mjpeg..
etc...
And more...
etc... 
And corrections (link a little blink at open time due to a window redraw
etc...


To day, some limits appears due to missing TLS/SSL support under ffmpeg   

Due to all changes I do, I had to run many time ffmpeg with several reboot making me searching why and write a quick code to load module et prevent reboot. Now, the only consuming storage is WPS (a WPS restart give back some storage but not all. Of course, no more reboot since I load module permanent (thanks to Dave pointing me to this)

Hope to provide quickly good new and update on the ffmpeghigh.cmd

Best regards
Remy   

[Remy]

I spent the time since first update to upgrade and correct latest finding in the code.
Now, both are at the same upgraded code level.
I hope it is better now.

Available on my web page.
Regards
Rémy