Troubleshooting

The following provides troubleshooting tips to help you get back on track and includes solutions or workarounds for common errors.

Debug Logging

To troubleshoot unexpected behavior in SignServer, enable more detailed (debug) logging by running the following command in the JBoss CLI:

Start JBoss CLI:
```
APPSRV_HOME/bin/jboss-cli.sh -c
```
CODE

Run the following commands to enable DEBUG logging:

/subsystem=logging/logger=org.ejbca:write-attribute(name=level, value=DEBUG)
/subsystem=logging/logger=org.cesecore:write-attribute(name=level, value=DEBUG)
/subsystem=logging/logger=org.signserver:write-attribute(name=level, value=DEBUG)

CODE

General Issues

Transaction rolled back error

A transaction timeout can occur for requests that are taking longer time to finish than what is configured for database transactions in the application server.

Example of errors displayed in the log:

13:21:17,483 ERROR [org.jboss.as.ejb3.invocation] (http-/0.0.0.0:8443-1) JBAS014134: EJB Invocation failed on component ProcessSessionBean for method public abstract org.signserver.common.data.Response org.signserver.ejb.interfaces.ProcessSessionLocal.process(org.signserver.server.log.AdminInfo,org.signserver.common.WorkerIdentifier,org.signserver.common.data.Request,org.signserver.common.RequestContext) throws org.signserver.common.IllegalRequestException,org.signserver.common.CryptoTokenOfflineException,org.signserver.common.SignServerException: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
...
Caused by: javax.transaction.RollbackException: JBAS014585: Transaction 'TransactionImple < ac, BasicAction: 0:ffff7f000001:672b819:5911a12e:f8 status: ActionStatus.ABORTED >' was already rolled back
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:99) [jboss-as-ejb3-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
    ... 100 more

TEXT

Possible reasons include large file uploads and/or downloads, and/or slow network connections, slow HSM connection or HSM, or that contacting external servers takes a long time (CRL/OCSP fetching or time-stamping etc).

Solutions

Since database transactions are not required if the worker is not configured to access the database, disabling the key usage counter, and not using any archivers will resolve this issue. For more information, see Limiting the number of signatures and Archiving.
Alternatively, if the key usage counter or an Archiver must be used, reconfigure the application server with a higher value for the transaction timeout. For more information, see Limiting the number of signatures and Archiving.

CKR_PIN_INCORRECT (Luna HSM)

When attempting to log into a Luna HSM, the HSM may require a PIN code from an external PIN pad. If you subsequently try to log in via an application, you might encounter a misleading exception.

ERROR [org.signserver.server.cryptotokens.PKCS11CryptoToken] (default task-2) Activate failed: Failed to initialize PKCS11 provider slot 'CAAUTHDEV'.: load failed: CKR_PIN_INCORRECT

CODE

A workaround we've seen could be to set ProtectedAuthenticationPathFlagStatus = 1 in the config file of the HSM

SQLFeatureNotSupportedException with PostgreSQL

When running SignServer with certain versions of PostgreSQL, an SQLFeatureNotSupportedException can occur on any database insert. By disabling the L2 and query cache you can forgo this issue. See Deploy-time Configuration.

P11NG / JackNJI11

UnsatisfiedLinkError

There can be a conflict between the JNA implementation in SignServer and the one installed in the system. This can be seen as errors similar to the following:

java.lang.UnsatisfiedLinkError: com.sun.jna.Native.malloc(J)J
java.lang.UnsatisfiedLinkError: Can't obtain static newInstance method for class com.sun.jna.Structure

The solution is to remove JNA from the system (i.e. apt-get remove libjna-java) or to run Java with:

-Djava.library.path=

CODE

USER_NOT_LOGGED_IN Error

If a USER_NOT_LOGGED_IN error is encountered while using the Utimaco HSM, the solution is to enable the KeepAlive setting in the Utimaco configuration file cs_pkcs11_R2.cfg:

# Prevents expiring session after inactivity of 15 minutes
KeepAlive = true

CODE

SunPKCS11 / PKCS11CryptoToken and Java 17

The PKCS11CryptoToken does not work on Java 17 unless the Java process that runs the application server is passed the JAVA_OPTS parameter "--add-exports=jdk.crypto.cryptoki/sun.security.pkcs11.wrapper=ALL-UNNAMED".

When using the PKCS11CryptoToken an error similar to the following may be logged:

19:54:50,047 ERROR [com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapper] (default task-1) Method sun.security.pkcs11.wrapper.PKCS11.CK_C_INITIALIZE_ARGS.getInstance was not accessible, this may be due to a change in the underlying library.: java.lang.IllegalAccessException: class com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapper cannot access class sun.security.pkcs11.wrapper.PKCS11 (in module jdk.crypto.cryptoki) because module jdk.crypto.cryptoki does not export sun.security.pkcs11.wrapper to unnamed module @64dbc7bd

CODE

SunPKCS11 / PKCS11CryptoToken and certain Java 17 versions

Certain Java 17 versions/builds attempt to call C_GetInfo before the call C_Initialize, resulting in many HSM drivers responding with CKR_CRYPTOKI_NOT_INITIALIZED.The issue has been observed with OpenJDK 17 version Red_Hat-17.0.6.0.10-1.fc36 and switching to a different Java version/build is the only known solution.

 12:30:33,581 ERROR [stderr] (default task-1) sunpkcs11: Initializing PKCS#11 library /opt/ETcpsdk/lib/linux-x86_64/libctsw.so
12:30:33,582 ERROR [stderr] (default task-1) sunpkcs11: Multi-threaded initialization failed: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_CRYPTOKI_NOT_INITIALIZED
12:30:33,582 ERROR [com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapper] (default task-1) Wrong arguments were passed to sun.security.pkcs11.wrapper.PKCS11.CK_C_INITIALIZE_ARGS.getInstance threw an exception for log.error(msg, e): java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapper.<init>(SunP11SlotListWrapper.java:144)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapperFactory.getInstance(SunP11SlotListWrapperFactory.java:74)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.SunP11SlotListWrapperFactory.getInstance(SunP11SlotListWrapperFactory.java:35)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.Pkcs11SlotLabel.getSlotListWrapper(Pkcs11SlotLabel.java:566)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.Pkcs11SlotLabel.getProvider(Pkcs11SlotLabel.java:120)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.Pkcs11SlotLabel.getP11Provider(Pkcs11SlotLabel.java:551)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.Pkcs11SlotLabel.getP11Provider(Pkcs11SlotLabel.java:516)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.P11Slot.<init>(P11Slot.java:57)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.P11Slot.getInstance(P11Slot.java:246)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.P11Slot.getInstance(P11Slot.java:203)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.pkcs11.P11Slot.getInstance(P11Slot.java:181)
	at deployment.signserver.ear//org.cesecore.keys.token.PKCS11CryptoToken.delayedInit(PKCS11CryptoToken.java:130)
	at deployment.signserver.ear//org.cesecore.keys.token.PKCS11CryptoToken.getP11slotWithDelayedInit(PKCS11CryptoToken.java:296)
	at deployment.signserver.ear//org.cesecore.keys.token.PKCS11CryptoToken.activate(PKCS11CryptoToken.java:153)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.BaseCryptoToken.autoActivate(BaseCryptoToken.java:115)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.BaseCryptoToken.getKeyStore(BaseCryptoToken.java:97)
	at deployment.signserver.ear//com.keyfactor.util.keys.token.BaseCryptoToken.getTokenStatus(BaseCryptoToken.java:655)
	at deployment.signserver.ear//org.signserver.server.cryptotokens.PKCS11CryptoToken.getCryptoTokenStatus(PKCS11CryptoToken.java:308)
	at deployment.signserver.ear//org.signserver.server.signers.BaseSigner.isCryptoTokenActive(BaseSigner.java:376)
	at deployment.signserver.ear//org.signserver.server.signers.BaseSigner.getStatus(BaseSigner.java:103)
	at deployment.signserver.ear//org.signserver.server.signers.CryptoWorker.getStatus(CryptoWorker.java:37)
	at deployment.signserver.ear.SignServer-ejb.jar//org.signserver.ejb.WorkerSessionBean.getStatus(WorkerSessionBean.java:287)
<...>
	at org.jboss.xnio@3.8.7.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_CRYPTOKI_NOT_INITIALIZED
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_GetInfo(Native Method)
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11$SynchronizedPKCS11.C_GetInfo(PKCS11.java:1681)
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.<init>(PKCS11.java:156)
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11$SynchronizedPKCS11.<init>(PKCS11.java:1668)
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.getInstance(PKCS11.java:194)
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.getInstance(PKCS11.java:166)
	... 248 more

CODE

SignClient

Cannot Produce CertificateVerify Signature

When using SignClient using a PKCS 11 token (e.g. a smart-card reader) for authenticating in some cases (e.g. both server and client running on Java 11) a cipher-suit-related error can occur:

ERROR [HTTPDocumentSigner] Failed sending request: Cannot produce CertificateVerify signature

CODE

As a workaround, SignClient can be forced to fall back to using TLS version 1.2 (as shown in the commented-out sample in script bin/signclient, a similar example is also present in the Windows batch file bin/signclient.cmd)

# In some cases, running SignClient authenticating with a P11 token
# (e.g. a smartcard reader) could give cipher-suit errors,
# In these cases, a workaround is to force the use of TLS version 1.2
JAVA_OPTS="$JAVA_OPTS  -Djdk.tls.client.protocols=TLSv1.2 -Dhttps.protocols=TLSv1.2"

CODE