Move the repetetive code into a small opensslrsa_valid_key_alg() helper function.
Closes #3785 (closed)