OPENSSL_cleanup fails to return all memory

The following discussions from !7417 (merged) should be addressed:

• @pemensik started a discussion: (+7 comments)

  But I found current 9.19.8 release does not compile on latest RHEL9 system with FIPS mode enabled. It crashes all programs:

  $ doc/misc/.libs/cfg_test --zonegrammar primary
  zone <string> [ <class> ] {
  	type primary;
  ...
  	zero-no-soa-ttl <boolean>;
  	zone-statistics ( full | terse | none | <boolean> );
  };
  ../../../lib/isc/mem.c:993: REQUIRE(((ctx) != ((void *)0) && ((const isc__magic_t *)(ctx))->magic == ((('M') << 24 | ('e') << 16 | ('m') << 8 | ('C'))))) failed, back trace
  /root/rpmbuild/BUILD/bind-9.19.8/build/doc/misc/../../lib/isc/.libs/libisc-9.19.8.so(+0x2e6c3)[0x7ffff7a2e6c3]
  /root/rpmbuild/BUILD/bind-9.19.8/build/doc/misc/../../lib/isc/.libs/libisc-9.19.8.so(isc_assertion_failed+0x10)[0x7ffff7a2e450]
  /root/rpmbuild/BUILD/bind-9.19.8/build/doc/misc/../../lib/isc/.libs/libisc-9.19.8.so(isc__mem_free+0x91)[0x7ffff7a46741]
  /usr/lib64/ossl-modules/fips.so(+0x15074)[0x7ffff6a0a074]
  /lib64/ld-linux-x86-64.so.2(+0x9f5e)[0x7ffff7fd0f5e]
  /lib64/libc.so.6(+0x574b5)[0x7ffff76574b5]
  /lib64/libc.so.6(on_exit+0x0)[0x7ffff7657630]
  /lib64/libc.so.6(+0x3feb7)[0x7ffff763feb7]
  /lib64/libc.so.6(__libc_start_main+0x80)[0x7ffff763ff60]

  in gdb:

  (gdb) bt
  #0  0x00007ffff76a154c in __pthread_kill_implementation () from /lib64/libc.so.6
  #1  0x00007ffff7654d46 in raise () from /lib64/libc.so.6
  #2  0x00007ffff76287f3 in abort () from /lib64/libc.so.6
  #3  0x00007ffff7a2e455 in isc_assertion_failed (file=<optimized out>, line=<optimized out>, type=<optimized out>, cond=<optimized out>)
      at ../../../lib/isc/assertions.c:50
  #4  0x00007ffff7a46741 in isc__mem_free (ctx=<optimized out>, ptr=<optimized out>, flags=<optimized out>) at ../../../lib/isc/mem.c:993
  #5  0x00007ffff6a0a074 in cleanup () at providers/fips/self_test.c:170
  #6  0x00007ffff7fd0f5e in _dl_fini () at dl-fini.c:142
  #7  0x00007ffff76574b5 in __run_exit_handlers () from /lib64/libc.so.6
  #8  0x00007ffff7657630 in exit () from /lib64/libc.so.6
  #9  0x00007ffff763feb7 in __libc_start_call_main () from /lib64/libc.so.6
  #10 0x00007ffff763ff60 in __libc_start_main_impl () from /lib64/libc.so.6
  #11 0x0000555555555995 in _start ()

  It seems FIPS mode cleanup is different than normal OpenSSL. It works fine without FIPS mode. But does not even compile under it. Found when trying to test these changes can pass, but haven't even got so far. To be investigated later.

• @fabled started a discussion: (+1 comment)

  @fabled This might affect the PKCS#11 provider as well.

  To some extent yes. But especially the case if using SoftHSMv2 with OpenSSL backend. Then there is circular dependency OpenSSL -> pkcs11-provider -> SoftHSMv2 -> OpenSSL. And care is needed to make sure things are released in right order during exit.