Author Topic: OS/4 (technical details only)  (Read 156557 times)

David Graser

  • Hero Member
  • *****
  • Posts: 870
  • Karma: +84/-0
    • View Profile
Re: OS/4 (technical details only)
« Reply #90 on: March 07, 2019, 06:23:13 pm »
OK, I uploaded a version with AVX disabled if you'd like to test.

OK just tested and this version works good.  Went to all the problem web sites and they loaded, worked, and played just fine.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: OS/4 (technical details only)
« Reply #91 on: March 07, 2019, 07:04:11 pm »
OK, we have a workaround, namely building with --disable-AVX.

Now we're just left with the question of why our toolchain is not aligning these AVX instructions. It only seems to be a problem in libavcodec/x86/fft.asm.
One possibility is a weakness in the AOUT format. I vaguely remember problems getting things to work as can be seen by this from libavutil/x86/x86inc.asm
Code: [Select]
; aout does not support align=
; NOTE: This section is out of sync with x264, in order to
; keep supporting OS/2.
%macro SECTION_RODATA 0-1 16
    %ifidn __OUTPUT_FORMAT__,aout
        SECTION .text
    %elifidn __OUTPUT_FORMAT__,coff
        SECTION .text
    %elifidn __OUTPUT_FORMAT__,win32
        SECTION .rdata align=%1
    %elif WIN64
        SECTION .rdata align=%1
    %else
        SECTION .rodata align=%1
    %endif
%endmacro

IIRC, one problem is that aout only supports text segments and is quite primitive compared to newer object formats.
Perhaps I should port the whole build system to using obj (OMF32) format.

Background, aout is an ancient object format that was once common.
Our GCC was forked a long time back and still outputs aout object code. Generally we then use emxomf at link time to convert to native omf and use the system linker to create executables.
Currently in FFmpeg, nasm outputs aout, which is also converted to omf at link time.
We can also convert GCC object files at build time, which would also allow nasm to output omf or obj as nasm calls it.
Obj also needs the FLAT and USE32 parameters to create 32 bit code as it was originally a 16 bit format.

Problem is testing as I don't have AVX hardware.

OS4User

  • Sr. Member
  • ****
  • Posts: 406
  • Karma: +10/-0
    • View Profile
Re: OS/4 (technical details only)
« Reply #92 on: March 07, 2019, 07:34:56 pm »
Problem is testing as I don't have AVX hardware.

As indirect way you can describe ps_cos16_1  as Public  and always check its address and segment in map file. And, of course, we can ask David to test AVX ver with proper alignment (at least I hope). 

For some reason  ps_cos16_1 comes to CODE seg (.text ?)  but not to .rdata with proper alignment.
 

David Graser

  • Hero Member
  • *****
  • Posts: 870
  • Karma: +84/-0
    • View Profile
Re: OS/4 (technical details only)
« Reply #93 on: March 07, 2019, 07:40:19 pm »
Problem is testing as I don't have AVX hardware.

As indirect way you can describe ps_cos16_1  as Public  and always check its address and segment in map file. And, of course, we can ask David to test AVX ver with proper alignment (at least I hope). 

For some reason  ps_cos16_1 comes to CODE seg (.text ?)  but not to .rdata with proper alignment.

I will be glad to help anyway I can.
« Last Edit: March 07, 2019, 07:47:58 pm by David Graser »

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: OS/4 (technical details only)
« Reply #94 on: March 07, 2019, 11:12:21 pm »
OK, we have a workaround, namely building with --disable-AVX.

Now we're just left with the question of why our toolchain is not aligning these AVX instructions. It only seems to be a problem in libavcodec/x86/fft.asm.
One possibility is a weakness in the AOUT format. I vaguely remember problems getting things to work as can be seen by this from libavutil/x86/x86inc.asm
Code: [Select]
; aout does not support align=
; NOTE: This section is out of sync with x264, in order to
; keep supporting OS/2.
%macro SECTION_RODATA 0-1 16
    %ifidn __OUTPUT_FORMAT__,aout
        SECTION .text
    %elifidn __OUTPUT_FORMAT__,coff
        SECTION .text
    %elifidn __OUTPUT_FORMAT__,win32
        SECTION .rdata align=%1
    %elif WIN64
        SECTION .rdata align=%1
    %else
        SECTION .rodata align=%1
    %endif
%endmacro

IIRC, one problem is that aout only supports text segments and is quite primitive compared to newer object formats.
Perhaps I should port the whole build system to using obj (OMF32) format.

Background, aout is an ancient object format that was once common.
Our GCC was forked a long time back and still outputs aout object code. Generally we then use emxomf at link time to convert to native omf and use the system linker to create executables.
Currently in FFmpeg, nasm outputs aout, which is also converted to omf at link time.
We can also convert GCC object files at build time, which would also allow nasm to output omf or obj as nasm calls it.
Obj also needs the FLAT and USE32 parameters to create 32 bit code as it was originally a 16 bit format.

Problem is testing as I don't have AVX hardware.

Well, should that not be:
%macro SECTION_RODATA 0-1 32

(I know little about the nasm syntax).

I think we are ALREADY using OMF and not AOUT format. At least that's what the OS/2 port of nasm should do (and I used nasm myself and linked to code produced by the Watcom and VAC compilers).

« Last Edit: March 07, 2019, 11:15:50 pm by Lars »

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: OS/4 (technical details only)
« Reply #95 on: March 08, 2019, 02:44:05 am »
Well, as I said, GCC actually puts out aout. Currently the nasm output is getting converted at link time to OMF as fewer changes were needed, so more likely to be accepted upstream, as well as static libs that could be used in more places. Plus at the time I was having problems with the right syntax. I've continued to learn.
Anyways, I'll change configure and x86inc.asm. I think the correct change is to something like
Code: [Select]
section .TEXT32 align=%1 use32 class=CODE
group CGROUP TEXT32
Have to look at the map files to make sure we don't end up with too many segments as well.
edit, used to be part of a kludge for macho (Mac) which seems to have gone away. x86inc.asm is actually from X264, who refused to support OS/2. I should double check on KOMH's latest patches.
« Last Edit: March 08, 2019, 02:49:23 am by Dave Yeo »

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: OS/4 (technical details only)
« Reply #96 on: March 08, 2019, 05:46:00 am »
OK, one problem with going with CFLAGS=-Zomf (needed if nasm emits obj format) is configure uses nm https://en.wikipedia.org/wiki/Nm_%28Unix%29, https://sourceware.org/binutils/docs/binutils/nm.html in various places including in choosing how to mangle symbols (whether to add an underline prefix).
The configure snippet is
Code: [Select]
check_cc <<EOF || die "Symbol mangling check failed."
int ff_extern;
EOF
sym=$($nm $TMPO | awk '/ff_extern/{ print substr($0, match($0, /[^ \t]*ff_extern/)) }')
extern_prefix=${sym%%ff_extern*}
Ideas?

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: OS/4 (technical details only)
« Reply #97 on: March 08, 2019, 06:49:38 am »
1) I used
SEGMENT _TEXT CLASS=CODE PUBLIC USE32 ALIGN=% FLAT
for the segment definition (SEGMENT is equivalent to SECTION). When linking with either the VAC or the WATCOM linker it appears that those linkers will only honour (or prefer) the CLASS on combining segments. The actual segment name was irrelevant. In both cases the segment was combined with the default code segments that the VAC and WATCOM compilers generated.

2) what error are you getting on running nm ? If it is about a missing underscore or excessive underscore, maybe GCC has a switch to control generation of the underscore before a symbol or maybe it has a switch to choose the default calling convention (which will impact the generated symbol names). Fortunately, CDECL and SYSCALL seem to be interchangeable enough to use them either way (and only have an impact on the generated symbol name).
I hope there is some switch documentation for the OS/2 port of GCC ...

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: OS/4 (technical details only)
« Reply #98 on: March 08, 2019, 06:52:43 am »
Found this GCC switch

-fleading-underscore
-fno-leading-underscore

https://gcc.gnu.org/onlinedocs/gcc/Option-Summary.html

and NASM similar:
--prefix
--postfix

NM has switches to control the mangling:
--demangle[=STYLE]
--no-demangle

Maybe taking all this together helps solving the problem.


« Last Edit: March 08, 2019, 07:47:10 am by Lars »

OS4User

  • Sr. Member
  • ****
  • Posts: 406
  • Karma: +10/-0
    • View Profile
Re: OS/4 (technical details only)
« Reply #99 on: March 08, 2019, 07:48:41 am »
Well, as I said, GCC actually puts out aout. Currently the nasm output is getting converted at link time to OMF as fewer changes were needed

Dave, AFAIK  OMF does not support 32 bit alignment.

For test purpose you can replace:

Code: [Select]
SECTION_RODATA 32     in   fft.asm line 54
with
Code: [Select]
SECTION .rdata align=32 public use32 FLAT class=DATA

Try to get from nasm   coff or win32  format,  but I am not sure if linker will accept this.
(What linker do you use ?)   -   may be wl  will be required.


Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: OS/4 (technical details only)
« Reply #100 on: March 08, 2019, 07:53:46 am »
Hi Lars,
Objconv should work as an nm, https://www.agner.org/optimize/#objconv also has a good disassembler that can output in nasm format and supports AVX. Compiles and works fine with
Code: [Select]
g++ -o objconv -O2 -Zomf *cpp
Don't really like using something that needs extra install step but fine for experimenting.
Can use ATTRIBUTE to switch between cdecl and stdcall but without the underlines, Firefox would probably not be able to use the DLLs.
The error with nm was of course, "unrecognized file format" or such. No OMF support.
It's bedtime here, so after work tomorrow I'll see if I can get nasm to output 32 bit flat obj format code. Ideally with patches that will be accepted upstream and doesn't screw up other projects that use FFmpeg.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: OS/4 (technical details only)
« Reply #101 on: March 08, 2019, 07:58:08 am »
Well, as I said, GCC actually puts out aout. Currently the nasm output is getting converted at link time to OMF as fewer changes were needed

Dave, AFAIK  OMF does not support 32 bit alignment.

For test purpose you can replace:

Code: [Select]
SECTION_RODATA 32     in   fft.asm line 54
with
Code: [Select]
SECTION .rdata align=32 public use32 FLAT class=DATA

Try to get from nasm   coff or win32  format,  but I am not sure if linker will accept this.
(What linker do you use ?)   -   may be wl  will be required.

Interesting, I'm using wl which may accept coff along with omf (documentation says it will, but whether on OS/2?). Still have the problem of mixing GCC and NASM code so need CPPFLAGS=-Zomf. ideally need to support static libs as well.
More tomorrow.

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: OS/4 (technical details only)
« Reply #102 on: March 08, 2019, 08:08:05 am »
1) there is no problem in telling nasm to align a segment/section to a 16-byte or 32-byte boundary using the SECTION / SEGMENT directive. Also if it outputs OMF format, I am using that already. But of course you need to take care that all variables but the first (which is aligned to the requested alignment) in the segment are also aligned to 16-byte / 32-byte. And you have to do that by hand for all subsequent variables, for example by injecting zero bytes ...
nasm also supports the ALIGN directive in the code but then you would precede every variable with that directive ...

2) yes, nm.exe cannot handle OMF format. I just found that out ...

3) yes, you definitely can tell nasm to output 32-bit flat OMF format (FLAT is the keyword to use in the segment declaration). That's what I have been doing. And it linked perfectly well with VAC and WATCOM compilers' generated code. Since GCC also accepts OMF input, it should also link with GCC (or the linkers that it calls).

4) as OS4user suggested, you might also want to move the SECTION_RODATA to the data segment. Then it is not "read only" anymore but who cares ...

5) I guess you meant to say "syscall" instead of "stdcall". GCC for OS/2 also understands the _System keyword (instead of specifying via "attribute") for the syscall calling convention.

OS4User

  • Sr. Member
  • ****
  • Posts: 406
  • Karma: +10/-0
    • View Profile
Re: OS/4 (technical details only)
« Reply #103 on: March 08, 2019, 08:50:34 am »
Dave

I have done some testing for -f flag for nasm command line:

1) coff: does not support align=:
"test1.nasm:1: error: standard COFF does not support section alignment specification"

2) win32:  seams does not support class=  and my test var comes  to code seg
but align=32 works ok

3) aout:  does not support align=  and other directives

4) obj:  what is most surprising OBJ  supports align=256 !!! and var comes to proper data seg

wlink and wl work with 1) 2) 4)

Variant 4 for me most preferable.

So macro should be like this:

Code: [Select]
; aout does not support align=
; NOTE: This section is out of sync with x264, in order to
; keep supporting OS/2.
%macro SECTION_RODATA 0-1 16
    %ifidn __OUTPUT_FORMAT__,aout
        SECTION .text
    %elifidn __OUTPUT_FORMAT__,coff
        SECTION .text align=%1
    %elifidn __OUTPUT_FORMAT__,obj
        SECTION .DATA32 align=%1 public use32 FLAT class=DATA
    %elifidn __OUTPUT_FORMAT__,win32
        SECTION .rdata align=%1
    %elif WIN64
        SECTION .rdata align=%1
    %else
        SECTION .rodata align=%1
    %endif
%endmacro

add to nasm command line:  -f obj  (instead of -f aout)

SECTION_RODATA 256    in   fft.asm line 54 ( if you do not like warning  "OBJ format does not support alignment of 32: rounding up to 256")

This way you will  get proper  fft.obj  which can be directly given to wl without any conversion.
« Last Edit: March 08, 2019, 10:17:47 am by OS4User »

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: OS/4 (technical details only)
« Reply #104 on: March 08, 2019, 06:00:16 pm »
Using
Code: [Select]
%elif __OUTPUT_FORMAT__,obj
        SECTION .DATA32 align=%1 public use32 FLAT class=DATA

Build dies with
Code: [Select]
Error! E2021: size of segment text exceeds 64k by 808181 bytes
make: *** [libavcodec/avcode57.dll] Error 1

Will investigate this evening. Might have to replace all the text segments with a macro.
edit: There's 81 instances of SECTION .text just in libavcodec which I assume aren't getting the use32 FLAT directive.
« Last Edit: March 08, 2019, 06:09:40 pm by Dave Yeo »