OpenSSL error queue not cleaned
OpenSSL 3 uses a queue for errors (see e.g. https://subscription.packtpub.com/book/web-development/9781800560345/9/ch09lvl1sec65/understanding-the-openssl-error-queue) the memory for errors in this queue is allocated
via isc__tls_malloc_ex()
and freed in three cases:
- When the queue is emptied via
ERR_get_error_all()
and friends, eg viadst__openssl_toresult()
- Via
isc__tls_shutdown()
when the memory pool is destroyed viaisc_mem_destroy()
- Via
OPENSSL_cleanup()
on a clean shutdown during library unload
Some parts of the code clean that queue directly after an error happens, which relates to 1):
status = EVP_PKEY_fromdata(
ctx, retpkey, private ? EVP_PKEY_KEYPAIR : EVP_PKEY_PUBLIC_KEY,
params);
if (status != 1) {
DST_RET(dst__openssl_toresult2("EVP_PKEY_fromdata",
DST_R_OPENSSLFAILURE));
}
ret = ISC_R_SUCCESS;
When tracing the library via eg valgrind these leaks do not show due to 2) and 3).
For the following trace there does not seem to be any cleanup and therefore should be a memory leak when the code path is triggered repeatedly maybe even an OOM situation (which I assume would be hard to trigger since not much memory is allocated for each error queue entry)
==137103== 16 bytes in 1 blocks are still reachable in loss record 33 of 718
==137103== at 0x48407B4: malloc (vg_replace_malloc.c:381)
==137103== by 0x4B4A6A9: mallocx (jemalloc_shim.h:65)
==137103== by 0x4B4A858: mem_get (mem.c:304)
==137103== by 0x4B4BF3A: isc__mem_allocate (mem.c:785)
==137103== by 0x4B5FB11: isc__tls_malloc_ex (tls.c:129)
==137103== by 0x54F1AF5: CRYPTO_strdup (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
==137103== by 0x54A8167: ERR_set_debug (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
==137103== by 0x53AFACC: ASN1_get_object (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
==137103== by 0x53A9040: d2i_ASN1_OBJECT (in /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
==137103== by 0x4950D55: check_private (rdata.c:624)
==137103== by 0x49865A5: fromwire_rrsig (rrsig_46.c:345)
==137103== by 0x49ADF93: dns_rdata_fromwire (rdata.c:824)