Configure the application server according to instructions in the following sections.

 

Download WildFly

WildFly can either be downloaded as a zip package or customized using Galleon. We recommend you to use Galleon, since it only downloads the components you need, as opposed to using the zip package, which bundles all configurations and JBoss modules (EJBCA is only using a subset of all functionality in WildFly).

WildFly is usually put in the /opt folder on Linux systems, and using a symlink to the WildFly folder makes it easy to switch to a new version of WildFly by simply overriding the symlink.

Using the Zip Package

You need to use the Jakarta EE Full & Web Distribution package of WildFly when downloading the zip package. If you have not downloaded WildFly already, you can use the following command to do so:

Download and unpack the WildFly zip package

wget https://download.jboss.org/wildfly/24.0.1.Final/wildfly-24.0.1.Final.zip -O /tmp/wildfly-24.0.1.Final.zip
unzip -q /tmp/wildfly-24.0.1.Final.zip -d /opt/
ln -snf /opt/wildfly-24.0.1.Final /opt/wildfly
BASH

Using Galleon

Instead of downloading the WildFly zip package containing everything (all default server configurations and all JBoss modules), using Galleon tooling you can choose to only install the components needed by EJBCA.

Download Galleon

Galleon is provided as a zip package from GitHub which you can download and run without any installation.

Download Galleon

wget https://github.com/wildfly/galleon/releases/download/4.2.8.Final/galleon-4.2.8.Final.zip -O /tmp/galleon-4.2.8.Final.zip
unzip -q /tmp/galleon-4.2.8.Final.zip -d /tmp/
cd /tmp/galleon-4.2.8.Final/bin
BASH

Layers Used By EJBCA

The table below lists the Galleon layers used by EJBCA.

The core-tools layer will include the JBoss and Elytron CLI which comes in handy if the Elytron credential store or the standalone.xml configuration file need to be tweaked later. If you don't need this, exclude the core-tools and management layers and add elytron separately. bean-validation can be excluded as well, but is good to have, at least in staging environments. Picketbox-based web security is required for EJBCA to detect authentication using client certificates.

Name

Description

Dependencies

cdi

Support for Jakarta Contexts and Dependency Injection.

base-server
bean-validation (optional)

core-tools

Support for jboss-cli, add-user and elytron-tool launch scripts and configuration files.

management (optional)

datasources

Support for datasources.

transactions

deployment-scanner

Support for deployment directory scanning.

base-server

discovery

Support for discovery.

base-server

ee

Support for common functionality in the Jakarta EE platform.

jsonb (optional)
naming

ejb

Support for Jakarta Enterprise Beans, excluding the IIOP protocol.

ejb-lite
messaging-activemq
remoting
resource-adapters
undertow

io

Support for XNIO workers and buffer pools.

base-server

jaxrs

Support for JAXRS.

web-server

jpa

Support for JPA (using the latest WildFly supported Hibernate release).
Alternative: jpa-distributed

bean-validation (optional)
datasources

jsf

Support for Jakarta Server Faces.

bean-validation (optional)
cdi (optional)
web-server

logging

Support for the logging subsystem.

base-server

mail

Support for Jakarta Mail.

base-server
naming

management

Support for remote access to management interfaces secured using Elytron.

elytron

webservices

Support for Jakarta XML Web Services

ejb-lite (optional)
messaging-activemq (optional)
web-server

legacy-security

Support for legacy Picketbox-based web security.

naming
vault

Download and Configure WildFly Using Galleon

Galleon can download and extract WildFly to the directory specified using the install command. EJBCA is running on a standalone server, and only this server configuration is needed. For a typical installation, use the following CLI recipe:

Download and build WildFly using Galleon

./galleon.sh install wildfly:current#24.0.1.Final --dir=/opt/wildfly-24.0.1.Final --default-configs=standalone/standalone.xml --layers=cdi,core-tools,datasources,deployment-scanner,discovery,ee,-jsonb,ejb,io,jaxrs,jpa,jsf,logging,mail,management,webservices,legacy-security
ln -snf /opt/wildfly-24.0.1.Final /opt/wildfly  
BASH

When building EJBCA from source, you need to have bin/client/jboss-client.jar in your WildFly directory at compile time for the CLI tools to work. This jar file is not available when building WildFly using Galleon, but you can fetch it manually from the Maven Central Repository.

mkdir /opt/wildfly/bin/client
wget https://repo1.maven.org/maven2/org/wildfly/wildfly-client-all/24.0.1.Final/wildfly-client-all-24.0.1.Final.jar -O /opt/wildfly/bin/client/jboss-client.jar
CODE

Remove RESTEasy-Crypto

The application server can sometimes load its own version of Bouncy Castle, resulting in incompatibility and/or conflict issues.

The following displays an example log message in case of a conflict:

ERROR [StartupSingletonBean] BouncyCastle is not loaded by an EJBCA classloader, version conflict is likely

A typical error occurring as a consequence of the wrong Bouncy Castle being loaded in EJBCA is:

ClassCastException setting BagAttributes, can not set friendly name: : java.lang.ClassCastException: org.bouncycastle.jcajce.provider.asymmetric.x509.X509CertificateObject cannot be cast to org.bouncycastle.jce.interfaces.PKCS12BagAttributeCarrier

To avoid issues and prevent WildFly from loading the Bouncy Castle library included in the RESTEasy-Crypto module, rather than the library included in EJBCA, run the following:

sed -i '/.*org.jboss.resteasy.resteasy-crypto.*/d' /opt/wildfly/modules/system/layers/base/org/jboss/as/jaxrs/main/module.xml
rm -rf /opt/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-crypto/
CODE

Create a Custom Configuration

Replace /opt/wildfly/bin/standalone.conf with the following Jinja2 template.

/opt/wildfly/bin/standalone.conf

if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
     JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"
fi

if [ "x$JAVA_OPTS" = "x" ]; then
     JAVA_OPTS="-Xms{{ HEAP_SIZE }}m -Xmx{{ HEAP_SIZE }}m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m"
     JAVA_OPTS="$JAVA_OPTS -Dhttps.protocols=TLSv1.2,TLSv1.3"
     JAVA_OPTS="$JAVA_OPTS -Djdk.tls.client.protocols=TLSv1.2,TLSv1.3"
     JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
     JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS"
     JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
     JAVA_OPTS="$JAVA_OPTS -Djboss.tx.node.id={{ TX_NODE_ID }}"
     JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
     JAVA_OPTS="$JAVA_OPTS -Djdk.tls.ephemeralDHKeySize=2048"
else
     echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi
BASH

Set Allowed Memory Usage

By default, 512 MB of heap (RAM) is allowed to be used by the application server. This is not sufficient to run EJBCA. We recommend to allocate at least 2048 MB of RAM. To increase the default value, run the following command:

sed -i -e 's/{{ HEAP_SIZE }}/2048/g' /opt/wildfly/bin/standalone.conf
BASH

Set the Transaction Node ID

Set the transaction node ID to a unique number. The node ID is used by the transactions subsystem and ensures that the transaction manager only recovers branches which match the specified identifier. It is imperative that this identifier is unique between WildFly instances sharing either an object store or access common resource managers (i.e. when multiple EJBCA instances access the same database).

sed -i -e "s/{{ TX_NODE_ID }}/$(od -A n -t d -N 1 /dev/urandom | tr -d ' ')/g" /opt/wildfly/bin/standalone.conf
BASH

Configure WildFly as a Service

Modern Linux systems use systemd to start and stop services. The WildFly zip package already contains the necessary files to run as a service, but they need to be installed manually. Once started as a service, WildFly will run as the wildfly user, and we need to add this user as well.

cp /opt/wildfly/docs/contrib/scripts/systemd/launch.sh /opt/wildfly/bin
cp /opt/wildfly/docs/contrib/scripts/systemd/wildfly.service /etc/systemd/system
mkdir /etc/wildfly
cp /opt/wildfly/docs/contrib/scripts/systemd/wildfly.conf /etc/wildfly
systemctl daemon-reload
useradd -r -s /bin/false wildfly
chown -R wildfly:wildfly /opt/wildfly-24.0.1.Final/
BASH

The docs directory does not exist when using Galleon. For reference, this is what the missing files look like:

/opt/wildfly/docs/contrib/scripts/systemd/launch.sh

#!/bin/bash
if [ "x$WILDFLY_HOME" = "x" ]; then
    WILDFLY_HOME="/opt/wildfly"
fi
if [[ "$1" == "domain" ]]; then
    $WILDFLY_HOME/bin/domain.sh -c $2 -b $3
else
    $WILDFLY_HOME/bin/standalone.sh -c $2 -b $3
fi
BASH

/opt/wildfly/docs/contrib/scripts/systemd/wildfly.service

[Unit]
Description=The WildFly Application Server
After=syslog.target network.target
Before=httpd.service

[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1
EnvironmentFile=-/etc/wildfly/wildfly.conf
User=wildfly
LimitNOFILE=102642
PIDFile=/var/run/wildfly/wildfly.pid
ExecStart=/opt/wildfly/bin/launch.sh $WILDFLY_MODE $WILDFLY_CONFIG $WILDFLY_BIND
StandardOutput=null

[Install]
WantedBy=multi-user.target
BASH

/opt/wildfly/docs/contrib/scripts/systemd/wildfly.conf

# The configuration you want to run
WILDFLY_CONFIG=standalone.xml

# The mode you want to run
WILDFLY_MODE=standalone

# The address to bind to
WILDFLY_BIND=0.0.0.0
BASH

Start WildFly

Start WildFly as a service.

systemctl start wildfly
BASH

Create an Elytron Credential Store

You can protect passwords by storing them in a credential store. The credential is encrypted with a master password which is fetched by WildFly on startup.

Create a Master Password

Create a script which outputs the master password to stdout and ensure the script can only be executed by the wildfly user.

echo '#!/bin/sh' > /usr/bin/wildfly_pass
echo "echo '$(openssl rand -base64 24)'" >> /usr/bin/wildfly_pass
chown wildfly:wildfly /usr/bin/wildfly_pass
chmod 700 /usr/bin/wildfly_pass
BASH

Create the Credential Store

Create a credential store in /opt/wildfly/standalone/configuration encrypted with the password echoed by the wildfly_pass script.

mkdir /opt/wildfly/standalone/configuration/keystore
chown wildfly:wildfly /opt/wildfly/standalone/configuration/keystore
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add(location=keystore/credentials, relative-to=jboss.server.config.dir, credential-reference={clear-text="{EXT}/usr/bin/wildfly_pass", type="COMMAND"}, create=true)'
BASH

Add Database Driver

For most database management systems, the JDBC driver can be added by hot-deploying it into the deployment directory. This will be picked up by WildFly and deployed so we can create a data source straight away. You can use a generic name, without version number, in order to get a generic driver-name for the data source command.

MariaDB

Download and deploy MariaDB JDBC driver

wget https://dlm.mariadb.com/1785291/Connectors/java/connector-java-2.7.4/mariadb-java-client-2.7.4.jar -O /opt/wildfly/standalone/deployments/mariadb-java-client.jar
BASH

PostgreSQL

Download and deploy PostgreSQL JDBC driver

wget https://jdbc.postgresql.org/download/postgresql-42.3.1.jar -O /opt/wildfly/standalone/deployments/postgresql-jdbc4.jar
BASH

Microsoft SQL Server

Download and deploy Microsoft SQL Server JDBC driver

wget https://github.com/microsoft/mssql-jdbc/releases/download/v12.2.0/mssql-jdbc-12.2.0.jre11.jar -O /opt/wildfly/standalone/deployments/mssql-jdbc.jre11.jar
BASH

Other Databases

If you are using another type of database, copy the JDBC driver to the deployment directory and make note of the driver class and driver-name shown in the server log, for later use when adding the data source. For example:

... INFO  [org.jboss.as.connector.deployers.jdbc] (...) WFLYJCA0005: Deploying non-JDBC-compliant driver class org.driver.class (version X.Y.Z)
... INFO  [org.jboss.as.connector.deployers.jdbc] (...) WFLYJCA0022: Started Driver service with driver-name = jdbc-driver.jar
TEXT

Add a Datasource

To add a datasource for EJBCA to use, run the commands in JBoss CLI according to the examples below.

Note that --driver-name is should be the same as the filename of the JAR file you copied in the step above.

The --jndi-name is is defined by the property datasource.jndi-name in database.properties, and the default value is used in the following example.

The --user-name and --password defines the username and password of the EJBCA database user. They correspond to the username and password defined by the properties database.username and database.password in database.properties. In production this password should be changed to a "real" password.

MariaDB

For MariaDB, run the following:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=dbPassword, secret-value="ejbca")'
/opt/wildfly/bin/jboss-cli.sh --connect 'data-source add --name=ejbcads --connection-url="jdbc:mysql://127.0.0.1:3306/ejbca" --jndi-name="java:/EjbcaDS" --use-ccm=true --driver-name="mariadb-java-client.jar" --driver-class="org.mariadb.jdbc.Driver" --user-name="ejbca" --credential-reference={store=defaultCS, alias=dbPassword} --validate-on-match=true --background-validation=false --prepared-statements-cache-size=50 --share-prepared-statements=true --min-pool-size=5 --max-pool-size=150 --pool-prefill=true --transaction-isolation=TRANSACTION_READ_COMMITTED --check-valid-connection-sql="select 1;"'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
XML

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

PostgreSQL

For PostgreSQL, run the following:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=dbPassword, secret-value="ejbca")'
/opt/wildfly/bin/jboss-cli.sh --connect 'data-source add --name=ejbcads --connection-url="jdbc:postgresql://127.0.0.1/ejbca" --jndi-name="java:/EjbcaDS" --use-ccm=true --driver-name="postgresql.jar" --driver-class="org.postgresql.Driver" --user-name="ejbca" --credential-reference={store=defaultCS, alias=dbPassword} --validate-on-match=true --background-validation=false --prepared-statements-cache-size=50 --share-prepared-statements=true --min-pool-size=5 --max-pool-size=150 --pool-prefill=true --transaction-isolation=TRANSACTION_READ_COMMITTED --check-valid-connection-sql="select 1;"'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
CODE

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Microsoft SQL Server

For Microsoft SQL Server, run the following:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=dbPassword, secret-value="ejbca")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=ejbcads:add(connection-url="jdbc:sqlserver://foobar.YOUR.DOMAIN:1433;DatabaseName=ejbca;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;sendStringParametersAsUnicode=false", min-pool-size=5, max-pool-size=150, jndi-name="java:/EjbcaDS", driver-name=mssql-jdbc.jre11.jar, user-name="ejbca", credential-reference={store=defaultCS, alias=dbPassword}, pool-prefill=false, pool-use-strict-min=false, idle-timeout-minutes=2)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
CODE

Microsoft SQL Server With Kerberos Authentication

Create a configuration file for Kerberos.

Replace YOUR.DOMAIN with your Windows domain and make sure foobar.YOUR.DOMAIN points to a valid server.

/etc/krb5.conf

# To opt out of the system crypto-policies configuration of krb5, remove the
# symlink at /etc/krb5.conf.d/crypto-policies which will not be recreated.
includedir /etc/krb5.conf.d/

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
dns_lookup_realm = true
dns_lookup_kdc = true
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
spake_preauth_groups = edwards25519
default_realm = YOUR.DOMAIN
default_tkt_enctypes = aes256-cts aes256-cts-hmac-sha1-96 aes128-cts
default_tgs_enctypes = aes256-cts aes256-cts-hmac-sha1-96 aes128-cts
permitted_enctypes = aes256-cts aes256-cts-hmac-sha1-96 aes128-cts

[realms]
YOUR.DOMAIN = {
    kdc = foobar.YOUR.DOMAIN
    default_domain = foobar.YOUR.DOMAIN
    admin_server = foobar.YOUR.DOMAIN
}

[domain_realm]
.your.domain = YOUR.DOMAIN
your.domain = YOUR.DOMAIN
BASH


Configure WildFly system properties.

/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=java.security.krb5.conf:add(value="/etc/krb5.conf")'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=sun.security.krb5.debug:add(value="false")'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=sun.security.spnego.debug:add(value="false")'
CODE

Set up Kerberos using Elytron.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/kerberos-security-factory=ejbca-db-krbsf:add(debug=false, principal=USER@YOUR.DOMAIN, path=wildfly.keytab, relative-to=jboss.server.config.dir, request-lifetime=-1, obtain-kerberos-ticket=true, server=false)
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/authentication-configuration=kerberos-conf:add(kerberos-security-factory=ejbca-db-krbsf)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/authentication-context=ejbca-ds-context:add(match-rules=[{authentication-configuration=kerberos-conf}])'
CODE

Create a datasource with a Kerberos authentication context.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=ejbcads:add(connection-url="jdbc:sqlserver://foobar.YOUR.DOMAIN:1433;DatabaseName=ejbca;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;sendStringParametersAsUnicode=false", min-pool-size=5, max-pool-size=150, jndi-name="java:/EjbcaDS", driver-name=mssql-jdbc.jre11.jar, elytron-enabled=true, authentication-context=ejbca-ds-context, allow-multiple-users=false, pool-prefill=false, pool-use-strict-min=false, idle-timeout-minutes=2)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
CODE

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Other Databases

If you are using another type of database, adapt the statement above with the correct --driver-name, --connection-url, --driver-class and --check-valid-connection-sql.

Configure WildFly Remoting

EJBCA needs to use JBoss Remoting for the EJBCA CLI to work. Configure it to use a separate port 4447 and remove any other dependency on remoting except for what EJBCA needs.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=remoting/http-connector=http-remoting-connector:write-attribute(name=connector-ref,value=remoting)'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=remoting:add(port=4447,interface=management)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=remoting:add(socket-binding=remoting,enable-http2=true)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure Logging

Configure logging in WildFly to be able to dynamically change logging while the application server is running.

Logging Configurations

Choose one of the logging configurations below.

Option 1 - Recommended Logging

INFO log level for org.ejbca, org.cesecore and com.keyfactor is recommended for production systems.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=com.keyfactor:add(level=INFO)'
BASH

Option 2 - Quiet Logging

If you prefer more quiet logging, configure WildFly to only log audit log messages, warnings and errors.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.audit.impl.log4j.Log4jDevice:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=com.keyfactor:add(level=WARN)'
BASH

Additional Logging Configuration

You may additionally want to add the following configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.jboss.as.config:write-attribute(name=level, value=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.jboss.as:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.wildfly:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.xnio:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.config.ConfigurationHolder:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.infinispan:add(level=WARN)'
BASH

Add Access Logging

To log all requests processed by the server, similar to the Apache access log, add the following configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/setting=access-log:add(pattern="%h %t \"%r\" %s \"%{i,User-Agent}\"", relative-to=jboss.server.log.dir, directory=access-logs)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=io.undertow.accesslog:add(level=INFO)'
CODE

Remove the Console Handler

Console logging is not used when running WildFly with systemd, and removing it can increase logging performance.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/root-logger=ROOT:remove-handler(name=CONSOLE)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/console-handler=CONSOLE:remove()'
CODE

Remove Old Log Files

Log files are rotated on a daily basis by default, but old log files are not deleted automatically. You can delete old log files using a cronjob:

/etc/cron.daily/remove-old-wildfly-logs.sh

#!/bin/sh
# Remove log files older than 7 days
find /opt/wildfly/standalone/log/ -type f -mtime +7 -name 'server.log*' -execdir rm -- '{}' \;
BASH

Make the file executable:

chmod +x /etc/cron.daily/remove-old-wildfly-logs.sh
CODE

Enable Syslog Shipping

To additionally send the server log over UDP to a syslog server such as Graylog or Logstash, configure a syslog handler in WildFly. Set the hostname and port of the syslog server, as well as the desired log level according to the following example:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/json-formatter=logstash:add(exception-output-type=formatted, key-overrides=[timestamp="@timestamp"],meta-data=[@version=1])'
/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=logging/syslog-handler=syslog-shipping:add(app-name=EJBCA,enabled=true,facility=local-use-0,hostname=$(hostname -f),level=INFO,named-formatter=logstash,port=514,server-address=syslog.server,syslog-format=RFC5424)"
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/root-logger=ROOT:add-handler(name=syslog-shipping)'
CODE

Enable Audit Logging To File

You can write the EJBCA audit log to a separate file. E.g. to log to /opt/wildfly/standalone/log/cesecore-audit.log, rotate every 128 MB and keep one rotated file:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/size-rotating-file-handler=cesecore-audit-log:add(file={path=cesecore-audit.log, relative-to=jboss.server.log.dir}, max-backup-index=1, rotate-size=128m)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.audit.impl.log4j.Log4jDevice:add'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.audit.impl.log4j.Log4jDevice:add-handler(name=cesecore-audit-log)'
CODE

Configure OCSP Logging

If you have enabled the OCSP audit log or the OCSP transaction log, configure an asynchronous logger in WildFly for best performance.

For the OCSP transaction log:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.certificates.ocsp.logging.TransactionLogger:add(use-parent-handlers=false)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.certificates.ocsp.logging.TransactionLogger:write-attribute(name=level, value=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/async-handler=ocsp-tx-async:add(queue-length="100")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/async-handler=ocsp-tx-async:write-attribute(name=level, value=DEBUG)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/async-handler=ocsp-tx-async:write-attribute(name="overflow-action", value="BLOCK")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.certificates.ocsp.logging.TransactionLogger:add-handler(name=ocsp-tx-async)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/periodic-rotating-file-handler=ocsp-tx:add(autoflush=true, append=true, suffix=".yyyy-MM-dd", file={path=ocsp-tx.log,relative-to=jboss.server.log.dir})'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/async-handler=ocsp-tx-async:add-handler(name=ocsp-tx)'
CODE

For the OCSP audit log:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.certificates.ocsp.logging.AuditLogger:add(use-parent-handlers=false)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.certificates.ocsp.logging.AuditLogger:write-attribute(name=level, value=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/async-handler=ocsp-audit-async:add(queue-length="100")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/async-handler=ocsp-audit-async:write-attribute(name=level, value=DEBUG)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/async-handler=ocsp-audit-async:write-attribute(name="overflow-action", value="BLOCK")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.certificates.ocsp.logging.AuditLogger:add-handler(name=ocsp-audit-async)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/periodic-rotating-file-handler=ocsp-audit:add(autoflush=true, append=true, suffix=".yyyy-MM-dd", file={path=ocsp-audit.log,relative-to=jboss.server.log.dir})'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/async-handler=ocsp-audit-async:add-handler(name=ocsp-audit)'
CODE

Archive rotated log files to save disk space using a cron job:

/etc/cron.daily/archive-rotated-ocsp-logs.sh

#!/bin/sh
# Compress the OCSP audit log and the OCSP transaction log after rotation
xz /opt/wildfly/standalone/log/ocsp-tx.log.*
xz /opt/wildfly/standalone/log/ocsp-audit.log.*
BASH

Make the file executable:

chmod +x /etc/cron.daily/archive-rotated-ocsp-logs.sh
CODE

HTTP(S) Configuration

The following section explains how to configure HTTP(S) using Undertow.

Remove Existing TLS and HTTP Configuration

Run the following commands in JBoss CLI to remove existing TLS and HTTP configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=default:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=https:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=http:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=https:remove()'
# These two lines are not needed if Galleon was used
/opt/wildfly/bin/jboss-cli.sh --connect '/core-service=management/security-realm=ApplicationRealm/server-identity=ssl:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
CODE

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Use 3-Port Separation

The following section explains how to set up Undertow with 3-port separation. Port 8080 is used for HTTP (unencrypted traffic), port 8442 for HTTPS (encrypted) traffic with only server authentication and port 8443 for HTTPS (encrypted) traffic with both server and client authentication.

Add New Interfaces and Sockets

To add new interfaces and sockets, use the following:

Consider binding to a specific interface instead of binding to all interfaces using 0.0.0.0.

/opt/wildfly/bin/jboss-cli.sh --connect '/interface=http:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/interface=httpspub:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/interface=httpspriv:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=http:add(port="8080",interface="http")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=httpspub:add(port="8442",interface="httpspub")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=httpspriv:add(port="8443",interface="httpspriv")'
BASH

Configure TLS

Configure TLS according to the following instructions.

Make sure the password of the keystore and truststore in this section are correct, in order for the deployment not to fail. In production the passwords should be changed to "real" passwords.

Note that TLS v1.3 is only available when running JDK 11 or greater.

Configure httpsserver.tokentype=P12 in the EJBCA configuration file web.properties.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsKeystorePassword, secret-value="serverpwd")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsTruststorePassword, secret-value="changeit")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKS:add(path="keystore/keystore.p12",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsKeystorePassword},type=PKCS12)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsTS:add(path="keystore/truststore.p12",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsTruststorePassword},type=PKCS12)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-manager=httpsKM:add(key-store=httpsKS,algorithm="SunX509",credential-reference={store=defaultCS, alias=httpsKeystorePassword})'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/trust-manager=httpsTM:add(key-store=httpsTS)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/server-ssl-context=httpspub:add(key-manager=httpsKM,protocols=["TLSv1.3","TLSv1.2"],use-cipher-suites-order=false,cipher-suite-filter="TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",cipher-suite-names="TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/server-ssl-context=httpspriv:add(key-manager=httpsKM,protocols=["TLSv1.3","TLSv1.2"],use-cipher-suites-order=false,cipher-suite-filter="TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",cipher-suite-names="TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256",trust-manager=httpsTM,need-client-auth=true)'
BASH

Add HTTP(S) Listeners

To add HTTP(S) listeners:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=http:add(socket-binding="http", redirect-socket="httpspriv")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=httpspub:add(socket-binding="httpspub", ssl-context="httpspub", max-parameters=2048)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=httpspriv:add(socket-binding="httpspriv", ssl-context="httpspriv", max-parameters=2048)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure the Firewall

Open port 8080, 8442 and 8443 for incoming TCP traffic.

Open ports in RHEL / CentOS

systemctl enable firewalld --now
firewall-cmd --set-default-zone=dmz
firewall-cmd --zone=dmz --permanent --add-port 8080/tcp
firewall-cmd --zone=dmz --permanent --add-port 8442/tcp
firewall-cmd --zone=dmz --permanent --add-port 8443/tcp
firewall-cmd --reload
CODE

Use 2-Port Separation

The following section explains how to set up Undertow with 2-port separation. Port 8080 is used for HTTP (unencrypted traffic) and port 8443 is used for HTTPS (encrypted) traffic with optional client authentication.

Add New Interfaces and Sockets

To add new interfaces and sockets, use the following:

Consider binding to a specific IP instead of binding to all interfaces using 0.0.0.0.

/opt/wildfly/bin/jboss-cli.sh --connect '/interface=http:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/interface=https:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=http:add(port="8080",interface="http")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=https:add(port="8443",interface="https")'
BASH

Configure TLS

Configure TLS according to the following instructions.

Make sure the password of the keystore and truststore in this section are correct, in order for the commands not to fail. In production the passwords should be changed to "real" passwords.

Note that TLS v1.3 is only available when running JDK 11 or greater.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsKeystorePassword, secret-value="serverpwd")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsTruststorePassword, secret-value="changeit")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKS:add(path="keystore/keystore.p12",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsKeystorePassword},type=PKCS12)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsTS:add(path="keystore/truststore.p12",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsTruststorePassword},type=PKCS12)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-manager=httpsKM:add(key-store=httpsKS,algorithm="SunX509",credential-reference={store=defaultCS, alias=httpsKeystorePassword})'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/trust-manager=httpsTM:add(key-store=httpsTS)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/server-ssl-context=https:add(key-manager=httpsKM,protocols=["TLSv1.3","TLSv1.2"],use-cipher-suites-order=false,cipher-suite-filter="TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",cipher-suite-names="TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256",trust-manager=httpsTM,want-client-auth=true,authentication-optional=true)'
BASH


Add HTTP(S) Listeners

To add HTTP(S) listeners:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=http:add(socket-binding="http", redirect-socket="https")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=https:add(socket-binding="https", ssl-context="https", max-parameters=2048)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure the Firewall

Open port 8080 and 8443 for incoming TCP traffic.

Open ports in RHEL / CentOS

systemctl enable firewalld --now
firewall-cmd --set-default-zone=dmz
firewall-cmd --zone=dmz --permanent --add-port 8080/tcp
firewall-cmd --zone=dmz --permanent --add-port 8443/tcp
firewall-cmd --reload
CODE

Alternatively, you may open port 80 and port 443 and redirect traffic to 8080 and 8443 respectively.

Set up NAT in RHEL / CentOS

systemctl enable firewalld --now
firewall-cmd --set-default-zone=dmz
firewall-cmd --zone=dmz --permanent --add-port 80/tcp
firewall-cmd --zone=dmz --permanent --add-port 443/tcp
firewall-cmd --permanent --add-forward-port=port=80:proto=tcp:toport=8080
firewall-cmd --permanent --add-forward-port=port=443:proto=tcp:toport=8443
firewall-cmd --reload
CODE

Use an HSM

You can store the private key for the TLS client certificate in an HSM instead for improved security and performance. WildFly contains the Sun PKCS11 provider which can be used to talk to the HSM using PKCS#11.

First, create a Sun PKCS11 configuration file according to the following Jinja2 template. Consult your HSM vendor for details on how to set PKCS#11 attributes.

/opt/wildfly/standalone/configuration/pkcs11.cfg

name = HSM
library = {{ PKCS11_LIBRARY }}
slot = {{ SLOT_NUMBER }}
attributes(*, CKO_PUBLIC_KEY, *) = {
  CKA_TOKEN = false
  CKA_ENCRYPT = false
  CKA_VERIFY = true
  CKA_WRAP = false
}
attributes(*, CKO_PRIVATE_KEY, *) = {
  CKA_TOKEN = true
  CKA_PRIVATE = true
  CKA_SENSITIVE = true
  CKA_EXTRACTABLE = false
  CKA_DECRYPT = false
  CKA_SIGN = true
  CKA_UNWRAP = false
  CKA_DERIVE = false
}
CODE

Make the file readable by WildFly.

chown wildfly:wildfly /opt/wildfly/standalone/configuration/pkcs11.cfg
CODE

Store the token PIN in your credential store.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=hsm,secret-value=123456)'
CODE

Create a provider loader and keystore for the Sun PKCS11 provider.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/provider-loader=pkcs11:add(class-names=[sun.security.pkcs11.SunPKCS11],path=pkcs11.cfg,relative-to=jboss.server.config.dir,module=sun.jdk)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKSInHSM:add(credential-reference={store=defaultCS, alias=hsm},type=PKCS11,providers=pkcs11)'
CODE

Generate a keypair and create a CSR using JBoss CLI. The CSR is stored in /opt/wildfly/standalone/configuration/keystore.

RSA key

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKSInHSM:generate-key-pair(alias=tlsKey0001, distinguished-name="CN=hostname", algorithm=RSA, key-size=2048)' 
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKSInHSM:generate-certificate-signing-request(alias=tlsKey0001, path=keystore/csr.pem, relative-to=jboss.server.config.dir, signature-algorithm=SHA256withRSA)
CODE

Issue the certificate from your favourite CA and put the certificate chain next to the CSR. Then install it on the token:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKSInHSM:import-certificate(alias=tlsKey0001, path=keystore/chain.pem, relative-to=jboss.server.config.dir)'
CODE

Switch to the HSM keystore.

/subsystem=elytron/key-manager=httpsKM:write-attribute(name=key-store,value=httpsKSInHSM)
/subsystem=elytron/key-manager=httpsKM:write-attribute(name=credential-reference,value={store=defaultCS, alias=hsm})
CODE

HTTP Protocol Behavior Configuration

The following shows an example HTTP protocol behavior configuration.

If you are using OCSP GET requests, setting URI encoding and allowing encoding for query and backslash are needed.

The commands below ensure the address in the EJBCA WS WSDL matches the hostname and port the client uses to access the server.

/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.catalina.connector.URI_ENCODING:add(value="UTF-8")'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.catalina.connector.USE_BODY_ENCODING_FOR_QUERY_STRING:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.tomcat.util.http.Parameters.MAX_COUNT:add(value=2048)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=webservices:write-attribute(name=wsdl-host, value=jbossws.undefined.host)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=webservices:write-attribute(name=modify-wsdl-address, value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Optional Configuration

The following sections cover optional configurations.

Remove Welcome Content

Removes the hardcoded welcome page in WildFly. This is not needed if Galleon was used.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/location="\/":remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=handler/file=welcome-content:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

You can also remove the actual files and save some disk space with:

rm -rf /opt/wildfly/welcome-content/
CODE

Redirect to Application for Unknown URLs

Known URLs for EJBCA starts with /ejbca, /crls, /certificates or /.well-known (EST and ACME) according to the following example:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/rewrite=redirect-to-app:add(redirect=true,target="/ejbca/")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=redirect-to-app:add(priority=1,predicate="method(GET) and not path-prefix(/ejbca,/crls,/certificates,/.well-known) and not equals({\%{LOCAL_PORT}, 4447})")'
BASH

Enable HTTP Strict Transport Layer Security

The HTTP Strict-Transport-Security response header (HSTS) defined in RFC 6797 tells the browser to only access the server using HTTPS. If you are not serving any resources on the domain over HTTP such as OCSP responses,  you can enable this to improve security. Start off with a small max-age value (such as 60 seconds) and then gradually increase.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/response-header=hsts:add(header-name="Strict-Transport-Security",header-value="max-age=31536000")'
CODE

You should also redirect any HTTP traffic to HTTPS. The example assumes port 80 is used for HTTP,  port 443 for HTTPS and that the server can be accessed using the domain name example.com. It is also assumed that there is optional client certificate authentication on port 443 similar to 2-port separation. In the case of 3-port separation, traffic should be redirected to port 8442.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/rewrite=http-to-https:add(redirect="true",target="https://example.com:443%U%q")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=http-to-https:add(predicate="equals(%p,80)")'
CODE

URL Rewriting

URL rewriting allows you to serve CRLs, CA certificates and OCSP responses on your own URLs. This can be done using rewrite rules in Undertow.

URL Rewriting for the CRL Store Servlet

Since EJBCA 7.2, the ability to customize the URL for the CRL store servlet at compile time has been removed. For more information, see ECA-7059 and the EJBCA 7.2 Upgrade Notes.

However, the same result can be achieved by configuring Undertow. As an example, to allow a client to download CRLs from the URL /ejbca/crls/ instead of /ejbca/publicweb/crls/, add the following configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/rewrite=crl-rewrite:add(target="/ejbca/publicweb/crls/$${1}")'
/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=undertow/server=default-server/host=default-host/filter-ref=crl-rewrite:add(predicate=\"method(GET) and regex('/crls/(.*)')\")"
BASH

Or if you want to link to a specific CRL directly, go to /ejbca/publicweb/crls/search.cgi, make a note of the appropriate iHash value, and then add a URL rewrite according to the following example:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/rewrite=cdp-RootCA:add(target="/ejbca/publicweb/crls/search.cgi?iHash=Aj2kgEB646IWuipyNwa970dIzOs")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=cdp-RootCA:add(predicate="method(GET) and path(/crl/RootCA.crl)")'
CODE

URL Rewriting for the Certstore Servlet

Since EJBCA 7.2, the ability to customize the URL for the Certstore servlet at compile time has been removed. For more information, see ECA-7059 and the EJBCA 7.2 Upgrade Notes.

However, the same result can be achieved by configuring Undertow. As an example, to allow a client to download CA certificates from the URL /ejbca/certificates/ instead of /ejbca/publicweb/certificates/, add the following configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/rewrite=certs-rewrite:add(target="/ejbca/publicweb/certificates/$${1}")'
/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=undertow/server=default-server/host=default-host/filter-ref=certs-rewrite:add(predicate=\"method(GET) and regex('/certificates/(.*)')\")"
BASH

Or if you want to link to a specific CA certificate directly, go to /ejbca/publicweb/certificates/search.cgi, make a note of the appropriate iHash value, and then add a URL rewrite according to the following example:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/rewrite=aia-RootCA:add(target="/ejbca/publicweb/certificates/search.cgi?iHash=Aj2kgEB646IWuipyNwa970dIzOs")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=aia-RootCA:add(predicate="method(GET) and path(/aia/RootCA.cer)")'
CODE

URL Rewriting for the OCSP Responder

All OCSP responses are served from the same URL in EJBCA. To access the OCSP responder on the somewhat more succinct /ocsp instead of /ejbca/publicweb/status/ocsp, reconfigure Undertow as follows:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/rewrite=rewrite-ocsp:add(target="/ejbca/publicweb/status/ocsp")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=rewrite-ocsp:add(predicate="path(/ocsp) and method(GET,POST)")'
CODE

Enable OCSP Revocation Checking

WildFly can check the validity of client certificates against the OCSP responder defined by the certificate's AIA extension:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/trust-manager=httpsTM:write-attribute(name=ocsp, value={})'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Remove the ExampleDS Datasource

You can remove the ExampleDS datasource as it is not being used. This is not needed if Galleon was used.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=ee/service=default-bindings:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect 'data-source remove --name=ExampleDS'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
CODE

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Remove Unused Subsystems and Extensions

Optionally remove the unused subsystems and extensions. This is not needed if Galleon was used.

If you used the Jakarta EE Full & Web Distribution zip package:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=jdr:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=sar:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=jmx:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=pojo:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-metrics-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-jwt-smallrye:remove()' 
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=ee-security:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-health-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-opentracing-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=distributable-web:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=security/security-domain=jaspitest:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/jdbc-driver=h2:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-config-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=request-controller:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=security-manager:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.config-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.metrics-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.jwt-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.clustering.web:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.health-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.opentracing-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.jboss.as.jdr:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.jboss.as.jmx:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.jboss.as.sar:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.jboss.as.pojo:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.ee-security:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.request-controller:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.security.manager:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
CODE

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Remove AJP

Clean up all AJP related configuration if you are not using it.

/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=ajp:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
CODE

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Enable AJP Connector

Enabling the AJP connector is only needed if you terminate the TLS connection at a proxy (such as Apache or nginx) running in front of WildFly:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/ajp-listener=ajp-listener:add(socket-binding=ajp, scheme=https, enabled=true)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Add a Request Limiter

You can limit the number of concurrent connections by adding a request limiter in Undertow. For example, to allow 100 connections to be processed at the same time, and allow up to 300 connections to be queued before new connections are rejected:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/request-limit=ejbca-request-limiter:add(max-concurrent-requests=100,queue-size=300)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=ejbca-request-limiter:add(predicate=path-prefix(/ejbca)'
CODE

Restrict Access to Services

You can whitelist IP addresses or block access to some services completely using predicates and handlers in Undertow.

For example, to only allow access to the CA web from localhost:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/expression-filter=local-only:add(expression="ip-access-control(acl={127.0.0.0/8 allow})")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=local-only:add(predicate="path-prefix(/ejbca/adminweb)")'
CODE

To block access to the public web pages and the EJBCA documentation:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/expression-filter=not-found:add(expression="response-code(404)")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=not-found:add(predicate="path-prefix(/ejbca/enrol,/ejbca/retrieve,/ejbca/inspect,/ejbca/doc) or path(/ejbca)")'
CODE

Add Support for Sending Email

To add support for sending email using an SMTP smarthost, configure EjbcaMail. Specify the hostname, address, port and SMTP credentials according to the following example:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=smtpPassword, secret-value="foo123")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=ejbca-mail-smtp:add(port="465", host="my.mail.server")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=mail/mail-session="java:/EjbcaMail":add(jndi-name=java:/EjbcaMail, from=noreply@ejbca.org)' 
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=mail/mail-session="java:/EjbcaMail"/server=smtp:add(outbound-socket-binding-ref=ejbca-mail-smtp, tls=true, username=smtpuser, credential-reference={store=defaultCS, alias=smtpPassword})'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

Only Deploy at Startup

Historically application servers are really bad at cleaning up memory from previous deployments and hot (re-)deploy is discouraged in production. To avoid manual deployment with the management interface, we can specify that the deployment directory should be scanned once at application server startup by setting the scan-interval to 0. This also prevents an attacker from loading a malicious JAR file by dropping it in the deployments directory.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=deployment-scanner/scanner=default:write-attribute(name=scan-interval,value=0)'
BASH

Increase the Deployment Timeout

If you are using HSMs with smart card authentication or if the database needs to reindex when WildFly boots, you may have to increase the deployment timeout to be able to deploy EJBCA correctly. The deployment timeout is specified in seconds, the command below sets it to 5 minutes.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=deployment-scanner/scanner=default:write-attribute(name=deployment-timeout,value=300)'
BASH

Disable Management Web Console

If you only plan on using the JBoss CLI anyway there is little reason to keep this around.

/opt/wildfly/bin/jboss-cli.sh --connect '/core-service=management/management-interface=http-interface:write-attribute(name=console-enabled,value=false)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Increase the Maximum Upload Size

Increase the maximum size of POST requests to 25 MB. This may be required if you need to upload large domain blocklists, zip packages with profiles, statedumps, etc. It may also be required for publishing relatively large CRLs to an external VA.

If 3-port separation is used, increase the maximum upload size for the httpspriv HTTPS listener.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=httpspriv:write-attribute(name=max-post-size,value=25485760)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

If 2-port separation is used, increase the maximum upload size for the https HTTPS listener instead.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=https:write-attribute(name=max-post-size,value=25485760)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
BASH

Enable WildFly Audit Logging

WildFly can audit log changes made to the management model and output audit log events in JSON format to the file /opt/wildfly/standalone/data/audit-log.log. Audit logging is disabled by default. To enable it, run:

/opt/wildfly/bin/jboss-cli.sh --connect '/core-service=management/access=audit/logger=audit-log:write-attribute(name=enabled,value=true)'
CODE

Remove WildFly Audit Logging

You may remove the WildFly audit logging configuration completely if you are not using it.

/opt/wildfly/bin/jboss-cli.sh --connect '/core-service=management/access=audit:remove()'
BASH

Configure Certificate Enrollment With ACME

The Elytron subsystem in WildFly can enroll for TLS server certificates using ACME. If you enroll from EJBCA, see EJBCA documentation on how to enable and configure ACME.

To prove ownership of the domain name in the CSR, WildFly must be able to serve an HTTP challenge over port 80 put in the server root directory. The easiest way to get this to work is to configure WildFly with 2-port separation.

  1. If the ACME server has a server certificate that chains up to an internal root not trusted by Java you need to manually add it to Java's truststore.

    Add a CA to the Java truststore in CentOS / RHEL

    keytool -importcert -noprompt -trustcacerts -alias internal-root -file InternalRoot.pem -keystore /etc/pki/java/cacerts -storepass changeit
    CODE
  2. Create a keystore where the keypair of the ACME account will be stored.

    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=accountsPassword, secret-value="foo123")'
    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=accountsKS:add(path=keystore/accounts.p12,relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=accountsPassword},type=PKCS12)'
    CODE
  3. Define the ACME directory to use. If you are enrolling directly against an EJBCA server, use something like https://my.ca/ejbca/acme/<alias>/directory.

    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/certificate-authority=AcmeCA:add(url="<ACME_DIRECTORY_URL>")'
    CODE
  4. Register an ACME account with the CA. The email address may be used by the CA to send announcements and warn about certificates that are about to expire.

    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/certificate-authority-account=AcmeAccount:add(certificate-authority=AcmeCA,alias=https,key-store=accountsKS,contact-urls=[mailto:admin@example.com])'
    CODE
  5. Enroll for a certificate to test the configuration. Replace <HOSTNAME> with the hostname of your machine. You can specify multiple hostnames by separating them with a comma.

    Enroll with an RSA keypair

    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKS:obtain-certificate(alias=https,domain-names=[<HOSTNAME1>],certificate-authority-account=AcmeAccount,agree-to-terms-of-service)'
    CODE

    Enroll with an EC keypair

    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKS:obtain-certificate(alias=https,domain-names=[<HOSTNAME1>],algorithm=EC,key-size=256,certificate-authority-account=AcmeAccount,agree-to-terms-of-service)'
    CODE
  6. Create a script to automatically renew the certificate when its about to expire.

    1. Install jq.

      Install jq in CentOS / RHEL

      yum install jq -y
      CODE
    2. Create the script called renew-wildfly-certificate.sh and put it in /etc/cron.daily to make it run on a daily basis. Adjust the parameters in the script to match your configuration.

      /etc/cron.daily/renew-wildfly-certificate.sh

      #!/bin/sh
      #
      #  Renew the WildFly TLS server certificate using ACME.
      #  Assumes an ACME account has been configured in Elytron.
      #  Requires 'jq' to be installed.
      #
      
      # The number of days before expiry where renewal should be attempted.
      days=30
      # Specify the type of keypair to generate. Possible values: [ "secp256r1", "RSA2048" ]
      key_specification="RSA2048"
      # The domain name(s) to put in the CSR. Separate with comma ','.
      domains="domain1.com,domain2.com"
      # The keystore alias of the certificate to renew
      alias="https"
      # The name of the ACME account in standalone.xml
      account="AcmeAccount"
      
      output=$(/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=elytron/key-store=httpsKS:should-renew-certificate(alias=$alias,expiration=$days)" --output-json)
      logger "$output"
      renew=$(echo "$output" | jq '.result["should-renew-certificate"]')
      if [ "$renew" = "true" ]; then
          logger "Attemping to renew domains [$domains]."
          if [ "$key_specification" = "secp256r1" ]; then
              output=$(/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=elytron/key-store=httpsKS:obtain-certificate(alias=$alias,domain-names=[$domains],algorithm=EC,key-size=256,certificate-authority-account=$account,agree-to-terms-of-service)" --output-json)
              logger "$output"
          fi
          if [ "$key_specification" = "RSA2048" ]; then
              output=$(/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=elytron/key-store=httpsKS:obtain-certificate(alias=$alias,domain-names=[$domains],certificate-authority-account=$account,agree-to-terms-of-service)" --output-json)
              logger "$output"
          fi
      fi
      BASH
    3. Make the script executable.

      chmod +x /etc/cron.daily/renew-wildfly-certificate.sh
      CODE

Start WildFly on System Boot

To make EJBCA available automatically after a system restart, use systemd.

systemctl enable wildfly
CODE

Performance Tuning

The following section covers WildFly configuration for maximizing EJBCA performance. The best setup could vary from installation to installation and should be manually fine-tuned. However, the "default" configuration would typically be a good starting point. You should also consider increasing the heap (RAM) for EJBCA instances under high load to at least 4 GB.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=io/worker=default/:write-attribute(name=task-core-threads,value=25)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=io/worker=default/:write-attribute(name=task-max-threads,value=100)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=io/worker=default/:write-attribute(name=io-threads,value=100)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=ejb3/strict-max-bean-instance-pool=slsb-strict-max-pool:undefine-attribute(name=derive-size)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=ejb3/strict-max-bean-instance-pool=slsb-strict-max-pool:write-attribute(name=max-pool-size, value=32)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=ejb3/thread-pool=default:write-attribute(name="max-threads", value=100)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=ejbcads/:write-attribute(name=max-pool-size,value=150)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'
CODE

(warning) Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Create a Snapshot of the Configuration

Create a snapshot of the current configuration to make sure you can revert back to a working state.

/opt/wildfly/bin/jboss-cli.sh --connect ':take-snapshot(name="Initial configuration")'
cp /opt/wildfly/standalone/configuration/standalone.xml /opt/wildfly/standalone/configuration/standalone.xml.backup
BASH

Create a Terminal Alias for Reading the Log File

EJBCA is logging all actions to the WildFly log file located in /opt/wildfly/standalone/log/server.log. It is often useful to be able to quickly filter and inspect the latest log lines. If you are using Fish you can add the following function to your config.fish:

function wflog --description 'Tail and optionally filter the WildFly log file'
    tail -f /opt/wildfly/standalone/log/server.log | \
        awk '/TRACE/ {print "\033[93m" $0 "\033[39m"}
             /DEBUG/ {print "\033[0;32m" $0 "\033[39m"}
             /INFO/ {print "\033[0;34m" $0 "\033[39m"}
             /WARN/ {print "\033[0;33m" $0 "\033[39m"}
             /ERROR/ {print "\033[0;31m" $0 "\033[39m"}
             /SEVERE/ {print "\033[1;31m" $0 "\033[39m"}
             !/(TRACE|DEBUG|INFO|WARN|ERROR|SEVERE)/ {print "\033[93m" $0 "\033[39m"}' | \
        grep --line-buffered --color=never -E "$argv[1]"
end
BASH

Or if you are using Bash, add this to your .bashrc:

wflog() { 
    tail -f /opt/wildfly/standalone/log/server.log | \
        awk '/TRACE/ {print "\033[93m" $0 "\033[39m"}
             /DEBUG/ {print "\033[0;32m" $0 "\033[39m"}
             /INFO/ {print "\033[0;34m" $0 "\033[39m"}
             /WARN/ {print "\033[0;33m" $0 "\033[39m"}
             /ERROR/ {print "\033[0;31m" $0 "\033[39m"}
             /SEVERE/ {print "\033[1;31m" $0 "\033[39m"}
             !/(TRACE|DEBUG|INFO|WARN|ERROR|SEVERE)/ {print "\033[93m" $0 "\033[39m"}' | \
        grep --line-buffered --color=never -E "$argv[1]"
}
BASH

Clear Command Line History

Clear the command line history to prevent accidental viewing of passwords put into the credential store.

history -c
BASH

Proxied SSL_CLIENT_CERT / Mutual Authentication

Proxied/forwared Mutual Authentication through a device like an F5 can be configured however the certificate must be in PEM format and including the header and footer.

References

The following lists links to references and useful external resources.

Next Step: Deploying EJBCA

For instructions on how to build and deploy EJBCA, see Deploying EJBCA.

Debug EJBCA

The following explains the configuration options in WildFly for troubleshooting EJBCA.

Switch to Debug Logging Globally

To enable debug logging globally, if you need to troubleshoot, follow these steps.

  1. Make a note of the current log level for the packages org.cesecore and org.ejbca.

    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:read-attribute(name=level)'
    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:read-attribute(name=level)'
    CODE
  2. Switch to debug logging.

    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:write-attribute(name=level, value=DEBUG)'
    /opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:write-attribute(name=level, value=DEBUG)'
    BASH

Once you are done troubelshooting, switch back to the log level you used previously. Note that having debug logging enabled globally will produce a lot of output, avoid using this configuration in production and consider enabling debugging for individual packages instead.

Switch to Debug Logging for Individual Packages

EJBCA is divided into different packages, and instead of raising the log level globally, you can be a bit more surgical in your approach and pick individual packages to troubleshoot. For available packages and classes, you can dynamically generate a list of JBoss CLI commands for enabling debug logging based on import statements and package declarations in the source code, with the most "popular" packages first:

grep -REh "^(package|import) (org.ejbca|org.cesecore|org.jitsi)" /opt/ejbca/modules | grep -v '*' | sed -E 's/(package|import) //' | sed -E 's/;\s*//' | awk '{n=split($0,c,".")};{for(i=1;i<=n;i++){for(j=1;j<=i;j++)printf("%s.",c[j]);printf("\n")}}' | sed -E 's/\.$//' | sort | uniq -c | sort -nr | awk '{if(NR>3) print "/opt/wildfly/bin/jboss-cli.sh --connect \'/subsystem=logging/logger="$2":add(level=DEBUG)\'"}'
CODE

If you want something more flashy, you can pipe the output to something like fzf, allowing you to filter and select the package or class name from a list. For example, add the following to your config.fish:

function wflogging --description 'Configure EJBCA logging in WildFly using JBoss CLI and fuzzy finding'
    grep -REh "^(package|import) (org.ejbca|org.cesecore|org.jitsi)" /opt/ejbca/modules | \
        grep -v '*' | \
        sed -E 's/(package|import) //' | \
        sed -E 's/;\s*//' | \
        awk '{n=split($0,c,".")};{for(i=1;i<=n;i++){for(j=1;j<=i;j++)printf("%s.",c[j]);printf("\n")}}' | \
        sed -E 's/\.$//' | \
        sort | uniq -c | sort -nr | \
        awk '{if(NR>3) print $2}' | \
        fzf --margin 1% --border=rounded --header 'Package / Class' \
            --bind "ctrl-a:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:add(level=INFO)')+refresh-preview" \
            --bind "ctrl-r:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:remove')+refresh-preview" \
            --bind "ctrl-t:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=TRACE)')+refresh-preview" \
            --bind "ctrl-d:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=DEBUG)')+refresh-preview" \
            --bind "ctrl-i:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=INFO)')+refresh-preview" \
            --bind "ctrl-w:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=WARN)')+refresh-preview" \
            --bind "ctrl-e:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=ERROR)')+refresh-preview" \
            --bind "alt-t:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect ':take-snapshot(name=wflogging_)')" \
            --bind "alt-r:execute(sudo mv /opt/wildfly/standalone/configuration/standalone.xml /opt/wildfly/standalone/configuration/standalone.xml.bak && sudo cp /opt/wildfly/standalone/configuration/standalone_xml_history/snapshot/wflogging_standalone.xml /opt/wildfly/standalone/configuration/standalone.xml && /opt/wildfly/bin/jboss-cli.sh --connect ':shutdown(restart=true)')" \
            --preview "printf \"Current logging configuration for {}\n\n\" && /opt/wildfly/bin/jboss-cli.sh --output-json --connect '/subsystem=logging/logger={}:read-resource'"
end
CODE

Or if you are using Bash, add this to your .bashrc:

wflogging() {
    grep -REh "^(package|import) (org.ejbca|org.cesecore|org.jitsi)" /opt/ejbca/modules | \
        grep -v '*' | \
        sed -E 's/(package|import) //' | \
        sed -E 's/;\s*//' | \
        awk '{n=split($0,c,".")};{for(i=1;i<=n;i++){for(j=1;j<=i;j++)printf("%s.",c[j]);printf("\n")}}' | \
        sed -E 's/\.$//' | \
        sort | uniq -c | sort -nr | \
        awk '{if(NR>3) print $2}' | \
        fzf --margin 1% --border=rounded --header 'Package / Class' \
            --bind "ctrl-a:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:add(level=INFO)')+refresh-preview" \
            --bind "ctrl-r:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:remove')+refresh-preview" \
            --bind "ctrl-t:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=TRACE)')+refresh-preview" \
            --bind "ctrl-d:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=DEBUG)')+refresh-preview" \
            --bind "ctrl-i:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=INFO)')+refresh-preview" \
            --bind "ctrl-w:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=WARN)')+refresh-preview" \
            --bind "ctrl-e:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger={}:write-attribute(name=level,value=ERROR)')+refresh-preview" \
            --bind "alt-t:execute-silent(/opt/wildfly/bin/jboss-cli.sh --connect ':take-snapshot(name=wflogging_)')" \
            --bind "alt-r:execute(sudo mv /opt/wildfly/standalone/configuration/standalone.xml /opt/wildfly/standalone/configuration/standalone.xml.bak && sudo cp /opt/wildfly/standalone/configuration/standalone_xml_history/snapshot/wflogging_standalone.xml /opt/wildfly/standalone/configuration/standalone.xml && /opt/wildfly/bin/jboss-cli.sh --connect ':shutdown(restart=true)')" \
            --preview "printf \"Current logging configuration for {}\n\n\" && /opt/wildfly/bin/jboss-cli.sh --output-json --connect '/subsystem=logging/logger={}:read-resource'"
}
CODE


This will add a command called wflogging, which opens fzf where you can select a class or package name in a list. Press Alt+T to take a snapshot of the existing log configuration, Ctrl + A to add logging, Ctrl + R to remove logging, and Ctrl + D to enable debugging for the selected item. The current configuration in the application server is shown in the preview window on the right.

Log Requests and Responses for the WebService API

To log the SOAP messages received by and sent from EJBCA:

/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.cxf.logging.enabled:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf.services:add(level=INFO)'
systemctl restart wildfly
CODE

To remove the configuration when you are done:

/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.cxf.logging.enabled:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf.services:remove'
systemctl restart wildfly
CODE

Log Requests and Responses for the EJBCA REST API

There is a servlet filter bundled with the EJBCA REST API which can be used to dump the full request and response.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca.ui.web.rest.api.config.RestLoggingFilter:add(level=TRACE)'
CODE

Log Requests and Responses for EST

There is a servlet filter bundled with the EJBCA REST API which can be used to dump the full request and response.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca.ui.web.protocol.EstLoggingFilter:add(level=TRACE)'
CODE

Log Requests for CMP

CmpMessageDispatcherSessionBean logs any CMP messages being sent from the CMP client on debug level. The whole ASN.1 is logged on trace level.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca.core.protocol.cmp.CmpMessageDispatcherSessionBean:add(level=TRACE)'
CODE

Log Requests and Responses for the ACME API

There is a servlet filter bundled with the EJBCA ACME implementation which can be used to dump the full request and response.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca.ui.web.protocol.acme.web.AcmeLoggingFilter:add(level=DEBUG)'
CODE

Enable Debugging for Microsoft Autoenrollment

To troubleshoot Microsoft Autoenrollment, enable debugging for the MSAE package.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca.msae:add(level=DEBUG)'
CODE

Monitor the Peer Systems Connections Pool

Each peer connector maintains a pool of TLS connections. Each request will occupy one connection from the pool, and requests are delayed if the pool is empty. If you suspect that more pooled connections are needed:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca.peerconnector.ra.PeerRaThrottleCounter:add(level=DEBUG)'
CODE

Debug Integration with Hardware Security Modules

P11Ng is the component in EJBCA responsible for communicating with the HSM using PKCS#11. If you need to troubleshoot HSM-related issues, you may enable debugging for P11Ng as follows:

Older installations may use the Sun PKCS#11 instead of P11Ng, in which case these instructions do not apply.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.keys.token.p11ng:add(level=DEBUG)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca.ui.p11ngcli:add(level=DEBUG)'
# Optionally add this logging item for in-depth PKCS#11 troubleshooting
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.keys.token.p11ng.provider.CryptokiDevice:add(level=TRACE)'
CODE

Enable Hibernate Statistics

Hibernate is the ORM library used by EJBCA to create SQL queries. You can enable Hibernate statistics to get the SQL query, as well as the time it takes to execute it, written in the server log.

Hibernate statistics are quite verbose and there may be database-specific tools better suited to troubleshoot database performance, however if you want to enable it:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=ejbcads/statistics=pool:write-attribute(name=statistics-enabled,value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=ejbcads/statistics=jdbc:write-attribute(name=statistics-enabled,value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.generate_statistics:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.show_sql:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.format_sql:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.use_sql_comments:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate.stat:add(level=DEBUG)'
systemctl restart wildfly
CODE

To remove the configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate.stat:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.use_sql_comments:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.format_sql:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.show_sql:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.generate_statistics:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=ejbcads/statistics=jdbc:write-attribute(name=statistics-enabled,value=false)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=ejbcads/statistics=pool:write-attribute(name=statistics-enabled,value=false)'
systemctl restart wildfly
CODE

Enable Profiling Using Glowroot

  1. Download Glowroot and put it in your WildFly directory.

    wget https://github.com/glowroot/glowroot/releases/download/v0.13.6/glowroot-0.13.6-dist.zip -O /tmp/glowroot.zip
    unzip -q /tmp/glowroot.zip -d /opt/wildfly
    chown -R wildfly:wildfly /opt/wildfly/glowroot
    CODE


  2. Register Glowroot as a Java agent and restart WildFly.

    sed -i '/-Djdk.tls.ephemeralDHKeySize=2048/ a \ \ \ JAVA_OPTS=\"$JAVA_OPTS -javaagent:/opt/wildfly/glowroot/glowroot.jar"' /opt/wildfly/bin/standalone.conf
    systemctl restart wildfly
    CODE
  3. Glowroot should now be available on http://localhost:4000.

EJBCA Services

This appendix lists the services exposed by EJBCA. This can be useful when configuring Undertow handlers or if you have an application layer firewall in front of WildFly.

ServiceProtocolURLComment
EJBCA CA webHTTPShttp[s]://{hostname}/ejbca/adminweb
EJBCA RA webHTTPShttp[s]://{hostname}/ejbca/ra
EJBCA Public web pagesHTTP, HTTPS

http[s]://{hostname}/ejbca
http[s]://{hostname}/ejbca/enrol
http[s]://{hostname}/ejbca/retrieve
http[s]://{hostname}/ejbca/inspect


Swagger UIHTTPShttps://{{hostname}}/ejbca/swagger-ui

Non-production mode only

EJBCA documentationHTTP, HTTPShttp[s]://{hostname}/ejbca/doc
EJBCA CRL distributionHTTP, HTTPShttp[s]://{hostname}/ejbca/publicweb/webdist/certdist?cmd=[crl|deltacrl]&issuer={subjectDn}
EJBCA CA certificate distributionHTTP, HTTPShttp[s]://{hostname}/ejbca/publicweb/certificates/search.cgi
EJBCA healthcheckHTTP, HTTPShttp://{hostname}/ejbca/publicweb/healthcheck/ejbcahealth
http://{hostname}/ejbca/publicweb/healthcheck/vastatus

EJBCA web service APIHTTPShttps://{hostname}/ejbca/ejbcaws/ejbcaws?wsdl
Microsoft AutoenrollmentHTTPShttps://{hostname}/ejbca/msae/CEPService?{alias}
EJBCA OCSP responderHTTP, HTTPShttp[s]://{hostname}/ejbca/publicweb/status/ocsp
EJBCA SCEPHTTP, HTTPShttp[s]://{hostname}/ejbca/publicweb/apply/scep/[{alias}/]pkiclient.exe
EJBCA CMPHTTP, HTTPShttp[s]://{hostname}/ejbca/publicweb/cmp[/{alias}]
EJBCA ACMEHTTP, HTTPShttp[s]://{hostname]/ejbca/acme/[{alias}/]
EJBCA ESTHTTPShttps://{hostname}/ejbca/.well-known/est/[{alias}/]
EJBCA REST APIHTTP, HTTPShttp[s]://{hostname}/ejbca/ejbca-rest-api
Peer systemsHTTPShttps://{hostname}/ejbca/peer/v1