Author Topic: Problems with OpenSSL library after update.  (Read 1230 times)

Digi

  • Jr. Member
  • **
  • Posts: 87
  • Karma: +4/-1
    • View Profile
Problems with OpenSSL library after update.
« on: January 24, 2026, 05:56:41 pm »
After updating the libraries on the internet server, one of the applications began crashing frequently. It turned out that the crashes were occurring in crypto11.dll when calling the SSL_accept() function (openssl ver. 1.1.1w-1).

I tried installing openssl version 3.4.1-2 from the netlabs-exp repository, and the situation repeated: crashes in crypto3.dll when calling SSL_accept().

Crashes don't happen every time. They mostly occur on certain connections from open port scanners or some hacking tools that operate over the internet.

My report on this issue on GitHub: https://github.com/bitwiseworks/openssl-os2/issues/6

If you use server applications related to openssl, do not update openssl yet!

Steven Levine

  • Full Member
  • ***
  • Posts: 114
  • Karma: +15/-0
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #1 on: January 25, 2026, 09:02:16 pm »
I recommend you install the debuginfo packages for the modules listed in the "Labels on the Stack" section.  This will make analyzing the exceptq report more efficient.

Also, you might want to consider installing http:/www.warpcave.com/betas/exceptq-7.11.6.2-shl-2023-12-31.zip.  It can help with understanding the call chain for binaries that do not use traditional stack frames.

Digi

  • Jr. Member
  • **
  • Posts: 87
  • Karma: +4/-1
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #2 on: January 26, 2026, 03:39:45 am »
Done. Updated the GitHub post.

Steven Levine

  • Full Member
  • ***
  • Posts: 114
  • Karma: +15/-0
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #3 on: January 29, 2026, 04:21:18 am »
The updated exceptq report helps a lot.  I'm going to chalk this trap up to bad luck.  Here's what I suspect is
happening.  We are trying to execute

 vmovdqa ymm7, [rbx+40h]

and exceptq tells us that

 EBX : read/exec  memory at 0001:00148930 in CRYPTO11

which confirms that ebx+40 is not properly aligned.  It also tells us that ebx is probably pointing at constant data stored in the code segment - possibly OPENSSL_ia32cap_P.  A copy of poly1305-x86.s would make this easier
to figure this out.

IAC, whatever the data is, it's now in an unaligned location in 1.1.1w.

To avoid this, we would need to figure out how to ensure the data is aligned and change to code to use vmovdqu rather than vmovdqa.

Digi

  • Jr. Member
  • **
  • Posts: 87
  • Karma: +4/-1
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #4 on: February 02, 2026, 01:45:23 am »
My research has shown that the problem only occurs when using the OS/4 kernel. This kernel, unlike the ArcaOS kernel, has support for AVX - NPX thread context switching, etc. Most likely, OpenSSL checks for AVX support (via xgetbv?), and the problematic section of code with the ArcaOS kernel simply does not execute.

It would be great if the alignment issue (?) were resolved. Or, if it were easier, AVX support could be removed from OpenSSL. Although that's certainly not the best solution.

Unfortunately, the ArcaOS kernel here does not work well in VMWare
« Last Edit: February 02, 2026, 10:11:56 pm by Digi »

Dave Yeo

  • Hero Member
  • *****
  • Posts: 5683
  • Karma: +145/-1
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #5 on: February 02, 2026, 02:07:45 am »
Can you ask the OS4 people for a switch to turn off the AVX handling? I forget the basics but the OS does advertise whether it supports AVX and that would all that needs to be turned off.

Digi

  • Jr. Member
  • **
  • Posts: 87
  • Karma: +4/-1
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #6 on: February 02, 2026, 03:25:08 am »
I think it's better to fix the problem itself than to create new workarounds. AVX support is a really big plus.

The OS/4 developers are currently at war in their own country ...[censored]... people there might have other concerns right now.
« Last Edit: February 02, 2026, 10:17:04 pm by Digi »

Dave Yeo

  • Hero Member
  • *****
  • Posts: 5683
  • Karma: +145/-1
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #7 on: February 02, 2026, 05:01:19 am »
OK, didn't realize where the developers are, pretty crappy situation. At that the whole world seems to be going down the flusher.
As for the problem, yes it should be fixed. There's always people with old hardware/operating systems vs developers with the latest and endless ram etc.

OS4User

  • Sr. Member
  • ****
  • Posts: 407
  • Karma: +10/-0
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #8 on: February 04, 2026, 05:29:59 pm »
It is a bit tricky case.

OS4Team added (may be 7 years ago or more) AVX support on user`s request and it works fine. Of course, the application which is supposed to use AVX has to be compiled and linked properly. It means all the data has to be aligned in accordance with the AVX requirements.

My investigation has shown than we can reach the necessary alignment but we have to provide the following:
 - compilation has to be done to ".elf"  (not ".obj")
 - the linker has to understand ".elf"

These two are very far from OS4Team interest and activity. What OS4Team can do to help  is only remove AVX support, but for me it sounds stupid.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 5683
  • Karma: +145/-1
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #9 on: February 04, 2026, 06:55:17 pm »
Yes, the long term plan is to move GCC to compiling to ELF, which it is already capable of. Wlink in theory can link ELF files to OS/2 executables and does mostly work for C programs. CPLUS is another matter. As well other parts of the toolchain would need updating. Need a skilled programmer that is interested.
As for OpenSSL, another option might be not using the assembly code. Haven't looked at OpenSSL, but generally these projects have C code as well as assembly with the C code being a fall back for times when there is no assembly available like using a different CPU. Be slower and need to recompile the library.

Steven Levine

  • Full Member
  • ***
  • Posts: 114
  • Karma: +15/-0
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #10 on: February 07, 2026, 06:55:23 am »
It occurred to me that we might not be the only platform that might bump up against this kind of compatibility problem, so I went hunting.

It appears it should be possible to configure OPENSSL at runtime to avoid attempting to use the problematic code via the OPENSSL_ia32cap environment variable.

See

  https://docs.openssl.org/1.1.1/man3/OPENSSL_ia32cap/#description

and

  https://stackoverflow.com/questions/40091888/how-to-disable-avx-instructions-in-openssl

For the curious, the relevant code is at

crypto\cryptlib.c:694
    if ((env = getenv("OPENSSL_ia32cap"))) {

in the netlabs openssl sources.

« Last Edit: February 07, 2026, 07:20:40 am by Steven Levine »

Digi

  • Jr. Member
  • **
  • Posts: 87
  • Karma: +4/-1
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #11 on: February 07, 2026, 11:48:28 am »
It appears it should be possible to configure OPENSSL at runtime to avoid attempting to use the problematic code via the OPENSSL_ia32cap environment variable.

I've been playing around with different values ​​for this variable, but so far without success.

For the curious, the relevant code is at

crypto\cryptlib.c:694
    if ((env = getenv("OPENSSL_ia32cap"))) {

in the netlabs openssl sources.

I don't see such a line in file https://github.com/bitwiseworks/openssl-os2/blob/master/crypto/cryptlib.c . There are only 282 lines.

Dave Yeo

  • Hero Member
  • *****
  • Posts: 5683
  • Karma: +145/-1
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #12 on: February 07, 2026, 05:41:38 pm »
Looking at the configure options, of interest,
Code: [Select]
# no-asm        do not use assembler
...
# 386           generate 80386 code in assembly modules
# no-sse2       disables IA-32 SSE2 code in assembly modules, the above
#               mentioned '386' option implies this one

Attempting to build the RPM, I've got to,
Code: [Select]
Configuring OpenSSL version 3.4.1 for target os2
Using os-specific seed configuration
syntax error at (eval 25) line 102, near ",
)"
Missing right curly or square bracket at (eval 25) line 104, at end of line
Created configdata.pm
Running configdata.pm
Can't locate OpenSSL/fallback.pm in @INC (@INC contains: K:/util/perl /@unixroot/usr/local/lib/perl5 /@unixroot/usr/local/share/perl5 /@unixroot/usr/lib/perl5/vendor_perl /@unixroot/usr/share/perl5/vendor_perl /@unixroot/usr/lib/perl5 /@unixroot/usr/share/perl5 .) at configdata.pm line 36239.
BEGIN failed--compilation aborted at configdata.pm line 36239.
error: Bad exit status from /@unixroot/var/tmp/rpm-tmp.ywakbN (%build)


RPM build errors:
    Bad exit status from /@unixroot/var/tmp/rpm-tmp.ywakbN (%build)

Need to continue adding Perl stuff and hopefully get around the syntax error.

Steven Levine

  • Full Member
  • ***
  • Posts: 114
  • Karma: +15/-0
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #13 on: Today at 04:51:55 am »
Oops - wrong repo and wrong branch.  In the current netlabs sources, the code has moved to crypto\cpuid.c.

Some of the interesting code points are at

crypto\cpuid.c:18
unsigned int OPENSSL_ia32cap_P[4];

crypto\cpuid.c:110
    if ((env = ossl_getenv("OPENSSL_ia32cap")) != NULL) {
        int off = (env[0] == '~') ? 1 : 0;

poly1305-x86.pl:119
      if ($avx>1) {
   &mov   ("ecx",&DWP(8,"edi"));
   &test   ("ecx",1<<5);         # AVX2?
   &jz   (&label("no_sse2"));

Since vmovdqa is an AVX2 instruction, we want to turn off

  bit #64+5 denoting availability of AVX2 instructions;

to disable the problematic code path.

The brute force

  set OPENSSL_ia32cap=:0

should do this.  The less brutish

  set OPENSSL_ia32cap=:~0x20

should also work, but might leave other problem code paths enabled.