Author Topic: Problems with OpenSSL library after update.  (Read 434 times)

Digi

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +4/-1
  • http://os2.snc.ru/
    • View Profile
    • OS/2 ports and applications
Problems with OpenSSL library after update.
« on: January 24, 2026, 05:56:41 pm »
After updating the libraries on the internet server, one of the applications began crashing frequently. It turned out that the crashes were occurring in crypto11.dll when calling the SSL_accept() function (openssl ver. 1.1.1w-1).

I tried installing openssl version 3.4.1-2 from the netlabs-exp repository, and the situation repeated: crashes in crypto3.dll when calling SSL_accept().

Crashes don't happen every time. They mostly occur on certain connections from open port scanners or some hacking tools that operate over the internet.

My report on this issue on GitHub: https://github.com/bitwiseworks/openssl-os2/issues/6

If you use server applications related to openssl, do not update openssl yet!

Steven Levine

  • Full Member
  • ***
  • Posts: 111
  • Karma: +15/-0
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #1 on: January 25, 2026, 09:02:16 pm »
I recommend you install the debuginfo packages for the modules listed in the "Labels on the Stack" section.  This will make analyzing the exceptq report more efficient.

Also, you might want to consider installing http:/www.warpcave.com/betas/exceptq-7.11.6.2-shl-2023-12-31.zip.  It can help with understanding the call chain for binaries that do not use traditional stack frames.

Digi

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +4/-1
  • http://os2.snc.ru/
    • View Profile
    • OS/2 ports and applications
Re: Problems with OpenSSL library after update.
« Reply #2 on: January 26, 2026, 03:39:45 am »
Done. Updated the GitHub post.

Steven Levine

  • Full Member
  • ***
  • Posts: 111
  • Karma: +15/-0
    • View Profile
Re: Problems with OpenSSL library after update.
« Reply #3 on: January 29, 2026, 04:21:18 am »
The updated exceptq report helps a lot.  I'm going to chalk this trap up to bad luck.  Here's what I suspect is
happening.  We are trying to execute

 vmovdqa ymm7, [rbx+40h]

and exceptq tells us that

 EBX : read/exec  memory at 0001:00148930 in CRYPTO11

which confirms that ebx+40 is not properly aligned.  It also tells us that ebx is probably pointing at constant data stored in the code segment - possibly OPENSSL_ia32cap_P.  A copy of poly1305-x86.s would make this easier
to figure this out.

IAC, whatever the data is, it's now in an unaligned location in 1.1.1w.

To avoid this, we would need to figure out how to ensure the data is aligned and change to code to use vmovdqu rather than vmovdqa.