OK, we have a workaround, namely building with --disable-AVX.
Now we're just left with the question of why our toolchain is not aligning these AVX instructions. It only seems to be a problem in libavcodec/x86/fft.asm.
One possibility is a weakness in the AOUT format. I vaguely remember problems getting things to work as can be seen by this from libavutil/x86/x86inc.asm
; aout does not support align=
; NOTE: This section is out of sync with x264, in order to
; keep supporting OS/2.
%macro SECTION_RODATA 0-1 16
%ifidn __OUTPUT_FORMAT__,aout
SECTION .text
%elifidn __OUTPUT_FORMAT__,coff
SECTION .text
%elifidn __OUTPUT_FORMAT__,win32
SECTION .rdata align=%1
%elif WIN64
SECTION .rdata align=%1
%else
SECTION .rodata align=%1
%endif
%endmacro
IIRC, one problem is that aout only supports text segments and is quite primitive compared to newer object formats.
Perhaps I should port the whole build system to using obj (OMF32) format.
Background, aout is an ancient object format that was once common.
Our GCC was forked a long time back and still outputs aout object code. Generally we then use emxomf at link time to convert to native omf and use the system linker to create executables.
Currently in FFmpeg, nasm outputs aout, which is also converted to omf at link time.
We can also convert GCC object files at build time, which would also allow nasm to output omf or obj as nasm calls it.
Obj also needs the FLAT and USE32 parameters to create 32 bit code as it was originally a 16 bit format.
Problem is testing as I don't have AVX hardware.