The Nitrokey HSM is very similar to the SmartCard-HSM and you use opensc-pkcs11 to manage the Nitrokey HSM from EJBCA. For installation instructions, refer to the Nitrokey HSM installation instructions. In the following example, opensc installed from the Nitrokey repository Nitrokey repository is used.

As of OpenJDK 11.0.17 and JDK 8u312, the Java PKCS#11 provider stopped working with Nitrokey/OpenSC for RSA keys. This is generic for all tokens using OpenSC as PKCS#11 driver. Using P11NG (Enterprise only) there are no issues.

This is due to checks that Java started doing on PKCS#11 mechanisms. Some issues reported in in OpenJDK related to this are: JDK-8176837 and JDK-8293345.
In short, it is due to the mechanismInfo for CKM_RSA_PKCS being reported back from OpenSC with "CKF_HW | CKF_DECRYPT | CKF_SIGN | CKF_VERIFY", while  Java requires that if it has CKF_DECRYPT it must also have CKF_ENCRYPT, otherwise the mechanism will be disabled.
You can also find information about it in this thread: https://stackoverflow.com/questions/70094324/migrating-from-jdk10-to-jdk11-sslconnection-ckr-key-type-inconsistent

You can make it work by configuring pkcs11.disableHashingSignMechanisms=false in conf/cesecore.properties.

The above issues are related to RSA keys only. EC keys work without issues.

Using Nitrokey HSM

After the installation you will be able to view the Nitrokey HSM:

user@linux:$ sc-hsm-tool 
Using reader with a card: Nitrokey Nitrokey HSM (DENK01018660000         ) 00 00
Version              : 3.1
Config options       :
  User PIN reset with SO-PIN enabled
SO-PIN tries left    : 15
User PIN tries left  : 3

user@linux:$ pkcs15-tool -D
Using reader with a card: Nitrokey Nitrokey HSM (DENK01018660000         ) 00 00
PKCS#15 Card [SmartCard-HSM]:
    Version        : 0
    Serial number  : DENK0101866
    Manufacturer ID: www.CardContact.de
    Flags          : 

PIN [UserPIN]
    Object Flags   : [0x3], private, modifiable
    Auth ID        : 02
    ID             : 01
<snip>

user@linux:$ pkcs11-tool -l -O --pin 648219
CODE

You can generate and test keys using P11NG-CLI.

./p11ng-cli.sh listslots --lib-file /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
./p11ng-cli.sh listobjects --lib-file /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so --slot-ref SLOT_NUMBER --slot 0
./p11ng-cli.sh signperformancetest --lib-file /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so --slot 0 --password 648219 --time-limit 5000 --alias ecKeysecp256r1 --signature-algorithm SHA256WithECDSA
./p11ng-cli.sh signperformancetest --lib-file /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so --slot 0 --password 648219 --time-limit 5000 --alias rsaKey2048 --signature-algorithm SHA256WithRSA
CODE

CientToolBox unfortunately does not honor the setting pkcs11.disableHashingSignMechanisms until EJBCA 8.0. For EC keys (and RSA after 8.0) you can generate and test keys with clientToolBox.

ant clientToolBox
cd dist/clientToolBox

./ejbcaClientToolBox.sh PKCS11HSMKeyTool generate /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so secp256r1 ecKeysecp256r1 0
<snip>

./ejbcaClientToolBox.sh PKCS11HSMKeyTool generate /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so 2048 rsaKey2048 0
<snip>
  ./ejbcaClientToolBox.sh PKCS11HSMKeyTool generate /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so 1024 testKey 0
Using Slot Reference Type: Slot Number.
PKCS11 Token [SunPKCS11-opensc-pkcs11.so-slot0] Password: 
2019-04-09 15:04:36,374 INFO  [org.cesecore.keys.util.SignWithWorkingAlgorithm] Signature algorithm 'SHA256WithRSA' working for provider 'SunPKCS11-opensc-pkcs11.so-slot0 version 10'.
Created certificate with entry testKey.

./ejbcaClientToolBox.sh PKCS11HSMKeyTool test /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so 0
Testing of key: testKey
Private part:
SunPKCS11-opensc-pkcs11.so-slot0 RSA private key, 1024 bits (id 140137944076096, token object, sensitive, unextractable)
RSA key:
  modulus: afc6f4149dc68d368a299cbf15370e36446bebc29770e35a98df974cf6ee033a180297cb6a4491b51e42135f2d5c5498e3ac5997c3c1c9af8d5a9881795c3715cbc330784964777321fcd3eb5c44dc6bdaa465a2f0d86fd6a509706ca5774a78b0b65b7f844231accfc73334664ad7255600dc0e9831578887fa3dab7051e3ed
  public exponent: 10001
encryption provider: SunJCE version 10; decryption provider: SunPKCS11-opensc-pkcs11.so-slot0 version 10; modulus length: 1024; byte length 117. The decoded byte string is equal to the original!
Signature test of key testKey: signature length 128; first byte 1f; verifying true
Signings per second: 5
Decryptions per second: 4
CODE

Using EJBCA, web.properties is pre-configured with the opensc-pkcs11 library named OpenSC as the PKCS#11 crypto token library.