PKCS#11 implementation of DH is broken (at least with SoftHSMv2)
The PKCS#11 implementation of DH algorithm doesn't work with SoftHSMv2 and although I very much doubt anybody uses TKEY or SIG(0) with HSMs, it will need to be fixed eventually.
The best way to test it is to checkout 328-make-openssl-mandatory
branch and revert the commit with commit message "Disable DH test with PKCS#11" and compile with ./configure --enable-developer --with-atf=/usr/local --with-openssl=/usr/local/opt/openssl --enable-native-pkcs11 --with-pkcs11=/usr/local/lib/softhsm/libsofthsm2.so
and the dh_test
in lib/dns/tests
fails with:
$ ./dh_test isc_dh_computesecret
failed: dh_test.c:65: ret != DST_R_COMPUTESECRETFAILURE
this is due to CKA_VALUE2
attribute is not found (https://gitlab.isc.org/isc-projects/bind9/blob/master/lib/dns/pkcs11dh_link.c#L127). I strongly suspect that (ab)using CKA_PRIVATE_EXPONENT for this might at fault:
* reuse CKA_PRIVATE_EXPONENT for key pair private value
[...]
#define CKA_VALUE2 CKA_PRIVATE_EXPONENT
but the PKCS#11 interface is written by cryptographers for cryptographers, so normal people can just blankly stare when reading the code using PKCS#11 API.
That said, the official specification for PKCS#11 Diffie-Hellman Section 2.4.4 is quite clear on the number of attributes, so my suggestion would be to follow the specification when fixing this.