OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Multimedia => Topic started by: Igor on February 18, 2021, 02:07:38 pm

Title: AVX safe FFmpeg
Post by: Igor on February 18, 2021, 02:07:38 pm
This is a version of ffmpeg compiled with AVX support and correct alignment. When launched with the OS / 4 kernel, it takes advantage of the AVX instructions of modern processors.
http://ru2.halfos.ru/ffmpeg/ffmpeg-4.2.2-os2-avx.zip (http://ru2.halfos.ru/ffmpeg/ffmpeg-4.2.2-os2-avx.zip)
Title: Re: AVX safe FFmpeg
Post by: Dariusz Piatkowski on February 18, 2021, 03:00:30 pm
Hi Igor,

Do you have a quick pre-requisites summary?

I ask b/c trying on non OS4 kernel (assuming here that it runs w/o the AVX instructions in such a scenario) shows the following:

Code: [Select]
[G:\test\ffmpeg_avx]ffmpeg
SYS1804: The system cannot find the file VORBIS0.

...so clearly a missing DLL.

Thanks!
Title: Re: AVX safe FFmpeg
Post by: Igor on February 18, 2021, 04:21:16 pm
Hi Igor,

Do you have a quick pre-requisites summary?

I ask b/c trying on non OS4 kernel (assuming here that it runs w/o the AVX instructions in such a scenario) shows the following:

Code: [Select]
[G:\test\ffmpeg_avx]ffmpeg
SYS1804: The system cannot find the file VORBIS0.

...so clearly a missing DLL.

Thanks!

Hi Dariusz,

Please install libvorbis via ANPM.
You can also use yum, but I don't know what the package is called there. Perhaps the same.
Title: Re: AVX safe FFmpeg
Post by: David Graser on February 18, 2021, 04:45:19 pm
Need to install OPUS in YUM also.
Title: Re: AVX safe FFmpeg
Post by: Dariusz Piatkowski on February 18, 2021, 05:29:00 pm
Igor,

Hi Igor,

Do you have a quick pre-requisites summary?

I ask b/c trying on non OS4 kernel (assuming here that it runs w/o the AVX instructions in such a scenario) shows the following:

Code: [Select]
[G:\test\ffmpeg_avx]ffmpeg
SYS1804: The system cannot find the file VORBIS0.

...so clearly a missing DLL.

Thanks!

Hi Dariusz,

Please install libvorbis via ANPM.
You can also use yum, but I don't know what the package is called there. Perhaps the same.

You know, I was a little too-quick on the response, should have done my homework and at least ran chkdll32x against the EXE.

So now that I've actually done that, here are the results:

Code: [Select]
[G:\test\ffmpeg_avx]chkdll32x ffmpeg.exe
Loading DLL 'libcx0' --> G:\USR\LIB\LIBCX0.DLL.
Loading DLL 'avdevi58' --> NOT loaded!
Unable to load DLL 'avdevi58'.  DosLoadModule returned: 2
OS/2 reports 'SDL2200' contributed to the failure.
Loading DLL 'avfilt7' --> NOT loaded!
Unable to load DLL 'avfilt7'.  DosLoadModule returned: 2
OS/2 reports 'VORBIS0' contributed to the failure.
Loading DLL 'avform58' --> NOT loaded!
Unable to load DLL 'avform58'.  DosLoadModule returned: 2
OS/2 reports 'VORBIS0' contributed to the failure.
Loading DLL 'avcode58' --> NOT loaded!
Unable to load DLL 'avcode58'.  DosLoadModule returned: 2
OS/2 reports 'VORBIS0' contributed to the failure.
Loading DLL 'avresa4' --> G:\TEST\FFMPEG_AVX\LIB\AVRESA4.DLL.
Loading DLL 'postpr55' --> G:\TEST\FFMPEG_AVX\LIB\POSTPR55.DLL.
Loading DLL 'swresa3' --> G:\TEST\FFMPEG_AVX\LIB\SWRESA3.DLL.
Loading DLL 'swscal5' --> G:\TEST\FFMPEG_AVX\LIB\SWSCAL5.DLL.
Loading DLL 'avutil56' --> G:\TEST\FFMPEG_AVX\LIB\AVUTIL56.DLL.
Loading DLL 'gcc1' --> G:\USR\LIB\GCC1.DLL.
Loading DLL 'libcn0' --> G:\USR\LIB\LIBCN0.DLL.
Loading DLL 'doscalls' --> loaded.

I'll YUM all of these to see which RPM packages they belong to...
Title: Re: AVX safe FFmpeg
Post by: Igor on February 18, 2021, 07:57:22 pm
Dariusz,

I have installed:
1. SDL2 2.0.12-2
2. libvorbis 1.3.7-2

Possibly needed packages SDL2_*
I also have them installed.
Title: Re: AVX safe FFmpeg
Post by: Dave Yeo on February 18, 2021, 08:04:12 pm
Here's the xqs files for it. The *_g.exe files aren't needed, they should have been the unstripped versions with debug data but seem the same as the *.exe versions.
Dmitriy looked at why -fno-common works here, https://github.com/bitwiseworks/gcc-os2/issues/11#issuecomment-759738933 (https://github.com/bitwiseworks/gcc-os2/issues/11#issuecomment-759738933) along with the following comments.
Only needed SDL2 2.0.12-2 here, already had libvorbis. The other SDL2 files will be handy for other projects.
Edit: fix zip name
Title: Re: AVX safe FFmpeg
Post by: Lars on February 19, 2021, 01:05:26 pm
Hi Ivan,

can you confirm that this AVX enabled version will ONLY EVER run on the OS/4 kernel ?

As far as I know, it requires OS support to save and restore the full 256 bit registers on a task switch and I seem to remember that the standard OS/2 kernel only supports SSE, that is, save and restore of only the 128 bit registers.
By the way, does anyone know if the 128 bits that SSE uses are only the lower 128 bits of the 256 bit registers or are these completely distinct register sets (one complete register set for SSE and an additional register set for AVX) ?

Or do I miss something altogether ?
Title: Re: AVX safe FFmpeg
Post by: OS4User on February 19, 2021, 01:32:20 pm
can you confirm that this AVX enabled version will ONLY EVER run on the OS/4 kernel ?

The R3 code should ask if the operating system supports AVX.  If yes - R3 may decides to use AVX code.  I mean, that correctly  compiled and linked dll  will work  on OS/2 and OS/4 as well.

In OS/4  dll will use AVX code, on OS/2 will NOT use AVX code.

Issue happened because     OS/4 supports AVX  and reports about it to R3 apps,  but FFmpeg libs were generated wrongly  and  cannot use their own AVX code (wrong data alignment).

Of course it does not affect pure OS/2 system - R3 apps   do not see AVX support from OS  and even do try   to use AVX code.

AFAIK,  Igor somehow  generated  dlls  with proper alignments -  so they can use  their AVX code,  but it does not affect OS/2 system at all.
Title: Re: AVX safe FFmpeg
Post by: Lars on February 19, 2021, 04:13:55 pm
I was aware of the fact that an application can check via CPUID instruction if the CPU supports AVX.
I was not aware of the fact that an application can check that the OS actually supports to save and restore the AVX register set (even though I remember that the OS is specifically required to set some flag in some register in order to indicate that it supports AVX save/restore registers).
Maybe I will have to look into the Intel / AMD specs ...
Title: Re: AVX safe FFmpeg
Post by: Igor on February 19, 2021, 04:24:59 pm
Dll kit to support streaming video Seamonkey and Firefox for OS / 4 kernel. Disabled AVX support. Failed to get this set of commands to work with the "-fno-common" key.
http://ru2.halfos.ru/ffmpeg/ffmpeg-os2-2.8.17-dll-noavx.zip (http://ru2.halfos.ru/ffmpeg/ffmpeg-os2-2.8.17-dll-noavx.zip)
Title: Re: AVX safe FFmpeg
Post by: Dave Yeo on February 19, 2021, 05:50:28 pm
Dll kit to support streaming video Seamonkey and Firefox for OS / 4 kernel. Disabled AVX support. Failed to get this set of commands to work with the "-fno-common" key.
http://ru2.halfos.ru/ffmpeg/ffmpeg-os2-2.8.17-dll-noavx.zip (http://ru2.halfos.ru/ffmpeg/ffmpeg-os2-2.8.17-dll-noavx.zip)

You might want to try linking with wl.exe (wlink) as it seems to me the linker handles larger alignment. The -fno-common is kind of a hack.
Title: Re: AVX safe FFmpeg
Post by: Dave Yeo on February 19, 2021, 06:06:51 pm
can you confirm that this AVX enabled version will ONLY EVER run on the OS/4 kernel ?

This FFmpeg plays videos fine on my system, CPU supports AVX, OS/2 kernel which doesn't.
Quote

As far as I know, it requires OS support to save and restore the full 256 bit registers on a task switch and I seem to remember that the standard OS/2 kernel only supports SSE, that is, save and restore of only the 128 bit registers.
By the way, does anyone know if the 128 bits that SSE uses are only the lower 128 bits of the 256 bit registers or are these completely distinct register sets (one complete register set for SSE and an additional register set for AVX) ?

Or do I miss something altogether ?

They're the same registers, just renamed and larger. SSE still uses the lower half.
Good code is supposed to check for operating system support and only use AVX if supported. I have a faint memory that SSE was similar as Win95 and such didn't support saving the registers
Title: Re: AVX safe FFmpeg
Post by: OS4User on February 19, 2021, 07:45:28 pm
You might want to try linking with wl.exe (wlink) as it seems to me the linker handles larger alignment

Unfortunately, wlink/wl  does not have an option/command to set  "between parts alignment"  within object.
Title: Re: AVX safe FFmpeg
Post by: Igor on February 19, 2021, 08:06:29 pm
This FFmpeg plays videos fine on my system, CPU supports AVX, OS/2 kernel which doesn't.
Yes of course. So everything will be, if the kernel does not support AVX, then this code is not included and another algorithm works that does not require special alignment.
But if the kernel supports AVX (this is an OS4 kernel), then the AVX code branch will turn on and a trap will appear.
If you disable AVX support during compilation, then the code will work on both the standard kernel and OS4.

Title: Re: AVX safe FFmpeg
Post by: Dave Yeo on February 19, 2021, 08:56:04 pm
You might want to try linking with wl.exe (wlink) as it seems to me the linker handles larger alignment

Unfortunately, wlink/wl  does not have an option/command to set  "between parts alignment"  within object.

Does it need it? The version I built previously used wl for linking and seemed to align stuff correctly, though I guess maybe a coincidence.
Title: Re: AVX safe FFmpeg
Post by: OS4User on February 19, 2021, 10:53:46 pm
You might want to try linking with wl.exe (wlink) as it seems to me the linker handles larger alignment

Unfortunately, wlink/wl  does not have an option/command to set  "between parts alignment"  within object.

Does it need it? The version I built previously used wl for linking and seemed to align stuff correctly, though I guess maybe a coincidence.

Data for  AVX instruction can be situated in many places, for .ex  - stack, dynamically allocated memory, static vars, etc.

I am speaking about static variables now. Our last GCC is able generate  proper .obj files with necessary  alignments. But if final data object  in LX module  is linked from parts (from several files)    we can meet and we indeed meet   wrong alignment.

Compiler reflects in obj file a demand that  it has to be linked with previous file  with certain alignment    in case current file  contains  data with specific alignment.

But not in our case. Historically, GCC for OS/2  uses a.out  as object format. A.out  is not able to carry info about requested alignment  to linker   that is why  we meet  wrong alignment in final binary.

The  best solution  is  to use  ELF  format  for objects files  (it requested a lot of time  for coding and fixing  GCC for OS/2).    One more way  -  could be just request  from linker   to  link every data parts  with 64 bytes align -  but wlink does not have such option/command -  so no way here.



Title: Re: AVX safe FFmpeg
Post by: Dave Yeo on February 27, 2021, 09:01:24 am
There's a new version uploaded to netlabs-exp. To quote,
Quote
I've uploaded a new version 4.2.2-4 to netlabs-exp with -fno-common --disable-avx2 --disable-avx512. Note that I still use --disable-avx2 because of my current assumption that it is not safe to generate AVX2 code w/o a proper alignment support from GCC (see my comment above). I guess that it works for you now just by accident.

Closing this. Please feel free to reopen if the new RPMs don't work for you.
untested here.
Title: Re: AVX safe FFmpeg
Post by: Igor on March 01, 2021, 06:30:44 pm
Hi, Dave

This version doesn't work for me...

Failing Instruction
_____________________________________________________________________

1261D064 >DB 0xc5                   (c5)
1261D065  CLD                       (fc)
1261D066  SUB     [0x1261cdf0], AH  (2825 f0cd6112)
1261D06C  DB 0xc5                   (c5)

And yet my assemblies are working.

There's a new version uploaded to netlabs-exp. To quote,
Quote
I've uploaded a new version 4.2.2-4 to netlabs-exp with -fno-common --disable-avx2 --disable-avx512. Note that I still use --disable-avx2 because of my current assumption that it is not safe to generate AVX2 code w/o a proper alignment support from GCC (see my comment above). I guess that it works for you now just by accident.

Closing this. Please feel free to reopen if the new RPMs don't work for you.
untested here.
Title: Re: AVX safe FFmpeg
Post by: Igor on March 01, 2021, 06:39:05 pm
I believe that a successful build with AVX is an accident and I suggest disabling the AVX code until the link is fixed.
Title: Re: AVX safe FFmpeg
Post by: Hank Starr on October 11, 2021, 10:22:34 am
Hi...Naturally, Prime95 consequently chooses the most up to date guidance set augmentation, like AVX, AVX2, or even AVX-512. To change this conduct, Prime95 should be begun and totally shut down again once. This will make the local.txt document. In it, prohibitions are alloted a worth of 0, though the code way that will be tried is appointed a worth of 1. In case you're not clear regarding which SSE adaptation is upheld by your CPU, both can be set to 1. Prime95 will pick the right fallback.