WildFly 24, 26, and JBoss EAP 7.4

Application Server specific information for WildFly 24 and 26. Also applicable for JBoss EAP 7.4.

This section provides information on configuring your application server.

Download WildFly

WildFly can either be downloaded as a zip package or customized using Galleon. We recommend you to use Galleon, since it only downloads the components you need, as opposed to using the zip package, which bundles all configurations and JBoss modules (SignServer is only using a subset of all functionality in WildFly).

WildFly is usually put in the /opt folder on Linux systems, and using a symlink to the WildFly folder makes it easy to switch to a new version of WildFly by simply overriding the symlink.

Using the Zip Package

You need to use the Jakarta EE Full & Web Distribution package of WildFly when downloading the zip package. If you have not downloaded WildFly already, you can use the following command to do so:

Download and unpack the WildFly zip package

wget https://download.jboss.org/wildfly/26.1.3.Final/wildfly-26.1.3.Final.zip -O /tmp/wildfly-24.0.1.Final.zip
unzip -q /tmp/wildfly-24.0.1.Final.zip -d /opt/
ln -snf /opt/wildfly-24.0.1.Final /opt/wildfly

BASH

Using Galleon

Instead of downloading the WildFly zip package containing everything (all default server configurations and all JBoss modules), using Galleon tooling you can choose to only install the components needed by SignServer.

Download Galleon

Galleon is provided as a zip package from GitHub which you can download and run without any installation.

Download Galleon

wget https://github.com/wildfly/galleon/releases/download/4.2.8.Final/galleon-4.2.8.Final.zip -O /tmp/galleon-4.2.8.Final.zip
unzip -q /tmp/galleon-4.2.8.Final.zip -d /tmp/
cd /tmp/galleon-4.2.8.Final/bin

BASH

Layers Used By SignServer

The table below lists the Galleon layers used by SignServer.

The core-tools layer will include the JBoss and Elytron CLI which comes in handy if the Elytron credential store or the standalone.xml configuration file need to be tweaked later. If you don't need this, exclude the core-tools and management layers and add elytron separately. bean-validation can be excluded as well, but is good to have, at least in staging environments. Picketbox-based web security is required for SignServer to detect authentication using client certificates.

Name	Description	Dependencies
cdi	Support for Jakarta Contexts and Dependency Injection.	base-server bean-validation (optional)
core-tools	Support for jboss-cli, add-user and elytron-tool launch scripts and configuration files.	management (optional)
datasources	Support for datasources.	transactions
deployment-scanner	Support for deployment directory scanning.	base-server
discovery	Support for discovery.	base-server
ee	Support for common functionality in the Jakarta EE platform.	jsonb (optional) naming
ejb	Support for Jakarta Enterprise Beans, excluding the IIOP protocol.	ejb-lite messaging-activemq remoting resource-adapters undertow
io	Support for XNIO workers and buffer pools.	base-server
jaxrs	Support for JAXRS.	web-server
jpa	Support for JPA (using the latest WildFly supported Hibernate release). Alternative: jpa-distributed	bean-validation (optional) datasources
jsf	Support for Jakarta Server Faces.	bean-validation (optional) cdi (optional) web-server
logging	Support for the logging subsystem.	base-server
mail	Support for Jakarta Mail.	base-server naming
management	Support for remote access to management interfaces secured using Elytron.	elytron
webservices	Support for Jakarta XML Web Services	ejb-lite (optional) messaging-activemq (optional) web-server
legacy-security	Support for legacy Picketbox-based web security.	naming vault

Download and Configure WildFly Using Galleon

Galleon can download and extract WildFly to the directory specified using the install command. SignServer is running on a standalone server, and only this server configuration is needed. For a typical installation, use the following CLI recipe:

Download and build WildFly using Galleon

./galleon.sh install wildfly:current#26.1.3.Final --dir=/opt/wildfly-26.1.3.Final --default-configs=standalone/standalone.xml --layers=cdi,core-tools,datasources,deployment-scanner,discovery,ee,-jsonb,ejb,io,jaxrs,jpa,jsf,logging,mail,management,webservices
ln -snf /opt/wildfly-24.0.1.Final /opt/wildfly

BASH

Remove RESTEasy-Crypto

The application server can sometimes load its own version of Bouncy Castle, resulting in incompatibility and/or conflict issues.

A typical error occurring as a consequence of the wrong Bouncy Castle being loaded in SignServer is:

ClassCastException setting BagAttributes, can not set friendly name: : java.lang.ClassCastException: org.bouncycastle.jcajce.provider.asymmetric.x509.X509CertificateObject cannot be cast to org.bouncycastle.jce.interfaces.PKCS12BagAttributeCarrier

To avoid issues and prevent WildFly from loading the Bouncy Castle library included in the RESTEasy-Crypto module, rather than the library included in SignServer, run the following:

sed -i '/.*org.jboss.resteasy.resteasy-crypto.*/d' /opt/wildfly/modules/system/layers/base/org/jboss/as/jaxrs/main/module.xml
rm -rf /opt/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-crypto/

CODE

Create a Custom Configuration

Replace /opt/wildfly/bin/standalone.conf with the following Jinja2 template.

/opt/wildfly/bin/standalone.conf

if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
     JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"
fi

if [ "x$JAVA_OPTS" = "x" ]; then
     JAVA_OPTS="-Xms{{ HEAP_SIZE }}m -Xmx{{ HEAP_SIZE }}m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m"
     JAVA_OPTS="$JAVA_OPTS -Dhttps.protocols=TLSv1.2,TLSv1.3"
     JAVA_OPTS="$JAVA_OPTS -Djdk.tls.client.protocols=TLSv1.2,TLSv1.3"
     JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
     JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS"
     JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
     JAVA_OPTS="$JAVA_OPTS -Djboss.tx.node.id={{ TX_NODE_ID }}"
     JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
     JAVA_OPTS="$JAVA_OPTS -Djdk.tls.ephemeralDHKeySize=2048"
else
     echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi

BASH

Set Allowed Memory Usage

By default, 512 MB of heap (RAM) is allowed to be used by the application server. This is not sufficient to run SignServer. We recommend to allocate at least 2048 MB of RAM. To increase the default value, run the following command:

sed -i -e 's/{{ HEAP_SIZE }}/2048/g' /opt/wildfly/bin/standalone.conf

BASH

Set the Transaction Node ID

Set the transaction node ID to a unique number. The node ID is used by the transactions subsystem and ensures that the transaction manager only recovers branches which match the specified identifier. It is imperative that this identifier is unique between WildFly instances sharing either an object store or access common resource managers (i.e. when multiple SignServer instances access the same database).

sed -i -e "s/{{ TX_NODE_ID }}/$(od -A n -t d -N 1 /dev/urandom | tr -d ' ')/g" /opt/wildfly/bin/standalone.conf

BASH

Configure WildFly as a Service (Optional)

Modern Linux systems use systemd to start and stop services. The WildFly zip package already contains the necessary files to run as a service, but they need to be installed manually. Once started as a service, WildFly will run as the wildfly user, and you need to add this user as well.

The following provides a suggestion for setting up WildFly as a systemd service. Note that the example may need to be adapted according to the system you are using.

cp /opt/wildfly/docs/contrib/scripts/systemd/launch.sh /opt/wildfly/bin
cp /opt/wildfly/docs/contrib/scripts/systemd/wildfly.service /etc/systemd/system
mkdir /etc/wildfly
cp /opt/wildfly/docs/contrib/scripts/systemd/wildfly.conf /etc/wildfly
systemctl daemon-reload
useradd -r -s /bin/false wildfly
chown -R wildfly:wildfly /opt/wildfly-26.1.3.Final/

BASH

The docs directory does not exist when using Galleon. For reference, this is what the missing files look like:

/opt/wildfly/docs/contrib/scripts/systemd/launch.sh

#!/bin/bash
if [ "x$WILDFLY_HOME" = "x" ]; then
    WILDFLY_HOME="/opt/wildfly"
fi
if [[ "$1" == "domain" ]]; then
    $WILDFLY_HOME/bin/domain.sh -c $2 -b $3
else
    $WILDFLY_HOME/bin/standalone.sh -c $2 -b $3
fi

BASH

/opt/wildfly/docs/contrib/scripts/systemd/wildfly.service

[Unit]
Description=The WildFly Application Server
After=syslog.target network.target
Before=httpd.service

[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1
EnvironmentFile=-/etc/wildfly/wildfly.conf
User=wildfly
LimitNOFILE=102642
PIDFile=/var/run/wildfly/wildfly.pid
ExecStart=/opt/wildfly/bin/launch.sh $WILDFLY_MODE $WILDFLY_CONFIG $WILDFLY_BIND
StandardOutput=null

[Install]
WantedBy=multi-user.target

BASH

/opt/wildfly/docs/contrib/scripts/systemd/wildfly.conf

# The configuration you want to run
WILDFLY_CONFIG=standalone.xml

# The mode you want to run
WILDFLY_MODE=standalone

# The address to bind to
WILDFLY_BIND=0.0.0.0

BASH

Start WildFly

The following provides two examples for starting WildFly, one if you are using a systemd service in the background and alternatively for starting WildFly in a separate terminal for testing.

Start WildFly installed as a systemd service

To start WildFly installed as a systemd service:

systemctl start wildfly

BASH

Start WildFly in a separate terminal for testing

Alternatively, start WildFly in a separate terminal for quick testing:

/opt/wildfly/bin/standalone.sh

CODE

Create an Elytron Credential Store

You can protect passwords by storing them in a credential store. The credential is encrypted with a master password which is fetched by WildFly on startup.

Create a Master Password

Create a script which outputs the master password to stdout and ensure the script can only be executed by the wildfly user.

echo '#!/bin/sh' > /usr/bin/wildfly_pass
echo "echo '$(openssl rand -base64 24)'" >> /usr/bin/wildfly_pass
chown wildfly:wildfly /usr/bin/wildfly_pass
chmod 700 /usr/bin/wildfly_pass

BASH

Create the Credential Store

Create a credential store in /opt/wildfly/standalone/configuration encrypted with the password echoed by the wildfly_pass script.

mkdir /opt/wildfly/standalone/configuration/keystore
chown wildfly:wildfly /opt/wildfly/standalone/configuration/keystore
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add(location=keystore/credentials, relative-to=jboss.server.config.dir, credential-reference={clear-text="{EXT}/usr/bin/wildfly_pass", type="COMMAND"}, create=true)'

BASH

Add Database Driver

For most database management systems, the JDBC driver can be added by hot-deploying it into the deployment directory. This will be picked up by WildFly and deployed so we can create a data source straight away. You can use a generic name, without version number, in order to get a generic driver-name for the data source command.

MariaDB

Download and deploy MariaDB JDBC driver

wget https://dlm.mariadb.com/1785291/Connectors/java/connector-java-2.7.4/mariadb-java-client-2.7.4.jar -O /opt/wildfly/standalone/deployments/mariadb-java-client.jar

BASH

PostgreSQL

Download and deploy PostgreSQL JDBC driver

wget https://jdbc.postgresql.org/download/postgresql-42.3.1.jar -O /opt/wildfly/standalone/deployments/postgresql-jdbc4.jar

BASH

Add a Datasource

To add a datasource for SignServer to use, run the commands in JBoss CLI according to the examples below.

Note that --driver-name is should be the same as the filename of the JAR file you copied in the step above.

The --jndi-name is is defined by the property datasource.jndi-name in database.properties, and the default value is used in the following example.

The --user-name and --password defines the username and password of the SignServer database user.

MariaDB

For MariaDB, run the following:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=dbPassword, secret-value="signserver")'
/opt/wildfly/bin/jboss-cli.sh --connect 'data-source add --name=signserverds --connection-url="jdbc:mysql://127.0.0.1:3306/signserver" --jndi-name="java:/SignServerDS" --use-ccm=true --driver-name="mariadb-java-client.jar" --driver-class="org.mariadb.jdbc.Driver" --user-name="signserver" --credential-reference={store=defaultCS, alias=dbPassword} --validate-on-match=true --background-validation=false --prepared-statements-cache-size=50 --share-prepared-statements=true --min-pool-size=5 --max-pool-size=150 --pool-prefill=true --transaction-isolation=TRANSACTION_READ_COMMITTED --check-valid-connection-sql="select 1;"'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

XML

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

PostgreSQL

For PostgreSQL, run the following:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=dbPassword, secret-value="signserver")'
/opt/wildfly/bin/jboss-cli.sh --connect 'data-source add --name=signserverds --connection-url="jdbc:postgresql://127.0.0.1/signserver" --jndi-name="java:/SignServerDS" --use-ccm=true --driver-name="postgresql-jdbc4.jar" --driver-class="org.postgresql.Driver" --user-name="signserver" --credential-reference={store=defaultCS, alias=dbPassword} --validate-on-match=true --background-validation=false --prepared-statements-cache-size=50 --share-prepared-statements=true --min-pool-size=5 --max-pool-size=150 --pool-prefill=true --transaction-isolation=TRANSACTION_READ_COMMITTED --check-valid-connection-sql="select 1;"'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

CODE

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure WildFly Remoting

SignServer needs to use JBoss Remoting for the SignServer Admin CLI to work. Configure it to use a separate port 4447 and remove any other dependency on remoting except for what SignServer needs.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=remoting/http-connector=http-remoting-connector:write-attribute(name=connector-ref,value=remoting)'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=remoting:add(port=4447,interface=management)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=remoting:add(socket-binding=remoting,enable-http2=true)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

BASH

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure Logging

Configure logging in WildFly to be able to dynamically change logging while the application server is running.

Logging Configurations

Choose one of the logging configurations below.

Option 1 - Recommended Logging

INFO log level for org.signserver and org.cesecore etc. is recommended for production systems.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.signserver:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=com.keyfactor:add(level=INFO)'

BASH

Option 2 - Quiet Logging

If you prefer more quiet logging, configure WildFly to only log audit log messages, warnings and errors.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.audit.impl.log4j.Log4jDevice:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.signserver:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=com.keyfactor:add(level=WARN)'

BASH

Additional Logging Configuration

You may additionally want to add the following configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.jboss.as.config:write-attribute(name=level, value=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.jboss.as:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.wildfly:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.xnio:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.config.ConfigurationHolder:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.infinispan:add(level=WARN)'

BASH

Add Access Logging

To log all requests processed by the server, similar to the Apache access log, add the following configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/setting=access-log:add(pattern="%h %t \"%r\" %s \"%{i,User-Agent}\"", relative-to=jboss.server.log.dir, directory=access-logs)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=io.undertow.accesslog:add(level=INFO)'

CODE

Remove the Console Handler

Console logging is not used when running WildFly with systemd, and removing it can increase logging performance.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/root-logger=ROOT:remove-handler(name=CONSOLE)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/console-handler=CONSOLE:remove()'

CODE

Remove Old Log Files

Log files are rotated on a daily basis by default, but old log files are not deleted automatically. You can delete old log files using a cronjob:

/etc/cron.daily/remove-old-wildfly-logs.sh

#!/bin/sh
# Remove log files older than 7 days
find /opt/wildfly/standalone/log/ -type f -mtime +7 -name 'server.log*' -execdir rm -- '{}' \;

BASH

Make the file executable:

chmod +x /etc/cron.daily/remove-old-wildfly-logs.sh

CODE

Enable Syslog Shipping

To additionally send the server log over UDP to a syslog server such as Graylog or Logstash, configure a syslog handler in WildFly. Set the hostname and port of the syslog server, as well as the desired log level according to the following example:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/json-formatter=logstash:add(exception-output-type=formatted, key-overrides=[timestamp="@timestamp"],meta-data=[@version=1])'
/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=logging/syslog-handler=syslog-shipping:add(app-name=SignServer,enabled=true,facility=local-use-0,hostname=$(hostname -f),level=INFO,named-formatter=logstash,port=514,server-address=syslog.server,syslog-format=RFC5424)"
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/root-logger=ROOT:add-handler(name=syslog-shipping)'

CODE

Enable Audit Logging To File

You can write the SignServer audit log to a separate file. E.g. to log to /opt/wildfly/standalone/log/cesecore-audit.log, rotate every 128 MB and keep one rotated file:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/size-rotating-file-handler=cesecore-audit-log:add(file={path=cesecore-audit.log, relative-to=jboss.server.log.dir}, max-backup-index=1, rotate-size=128m)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.audit.impl.log4j.Log4jDevice:add'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.audit.impl.log4j.Log4jDevice:add-handler(name=cesecore-audit-log)'

CODE

HTTP(S) Configuration

The following section explains how to configure HTTP(S) using Undertow.

Remove Existing TLS and HTTP Configuration

Run the following commands in JBoss CLI to remove existing TLS and HTTP configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=default:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=https:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=http:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=https:remove()'
# These two lines are not needed if Galleon was used
/opt/wildfly/bin/jboss-cli.sh --connect '/core-service=management/security-realm=ApplicationRealm/server-identity=ssl:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

CODE

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Use 3-Port Separation

The following section explains how to set up Undertow with 3-port separation. Port 8080 is used for HTTP (unencrypted traffic), port 8442 for HTTPS (encrypted) traffic with only server authentication and port 8443 for HTTPS (encrypted) traffic with both server and client authentication.

Add New Interfaces and Sockets

To add new interfaces and sockets, use the following:

Consider binding to a specific interface instead of binding to all interfaces using 0.0.0.0.

/opt/wildfly/bin/jboss-cli.sh --connect '/interface=http:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/interface=httpspub:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/interface=httpspriv:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=http:add(port="8080",interface="http")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=httpspub:add(port="8442",interface="httpspub")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=httpspriv:add(port="8443",interface="httpspriv")'

BASH

Configure TLS

Configure TLS according to the following instructions.

Make sure the password of the keystore and truststore in this section are correct, in order for the deployment not to fail. In production the passwords should be changed to "real" passwords.

Note that TLS v1.3 is only available when running JDK 11 or greater.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsKeystorePassword, secret-value="serverpwd")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsTruststorePassword, secret-value="changeit")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKS:add(path="keystore/keystore.jks",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsKeystorePassword},type=JKS)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsTS:add(path="keystore/truststore.jks",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsTruststorePassword},type=JKS)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-manager=httpsKM:add(key-store=httpsKS,algorithm="SunX509",credential-reference={store=defaultCS, alias=httpsKeystorePassword})'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/trust-manager=httpsTM:add(key-store=httpsTS)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/server-ssl-context=httpspub:add(key-manager=httpsKM,protocols=["TLSv1.3","TLSv1.2"],use-cipher-suites-order=false,cipher-suite-filter="TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",cipher-suite-names="TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/server-ssl-context=httpspriv:add(key-manager=httpsKM,protocols=["TLSv1.3","TLSv1.2"],use-cipher-suites-order=false,cipher-suite-filter="TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",cipher-suite-names="TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256",trust-manager=httpsTM,need-client-auth=true)'

BASH

Add HTTP(S) Listeners

To add HTTP(S) listeners:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=http:add(socket-binding="http", redirect-socket="httpspriv")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=httpspub:add(socket-binding="httpspub", ssl-context="httpspub", max-parameters=2048)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=httpspriv:add(socket-binding="httpspriv", ssl-context="httpspriv", max-parameters=2048)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

BASH

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure the Firewall

Open port 8080, 8442 and 8443 for incoming TCP traffic.

Open ports in RHEL / CentOS

systemctl enable firewalld --now
firewall-cmd --set-default-zone=dmz
firewall-cmd --zone=dmz --permanent --add-port 8080/tcp
firewall-cmd --zone=dmz --permanent --add-port 8442/tcp
firewall-cmd --zone=dmz --permanent --add-port 8443/tcp
firewall-cmd --reload

CODE

Use 2-Port Separation

The following section explains how to set up Undertow with 2-port separation. Port 8080 is used for HTTP (unencrypted traffic) and port 8443 is used for HTTPS (encrypted) traffic with optional client authentication.

Add New Interfaces and Sockets

To add new interfaces and sockets, use the following:

Consider binding to a specific IP instead of binding to all interfaces using 0.0.0.0.

/opt/wildfly/bin/jboss-cli.sh --connect '/interface=http:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/interface=https:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=http:add(port="8080",interface="http")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=https:add(port="8443",interface="https")'

BASH

Configure TLS

Configure TLS according to the following instructions.

Make sure the password of the keystore and truststore in this section are correct, in order for the commands not to fail. In production the passwords should be changed to "real" passwords.

Note that TLS v1.3 is only available when running JDK 11 or greater.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsKeystorePassword, secret-value="serverpwd")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsTruststorePassword, secret-value="changeit")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKS:add(path="keystore/keystore.p12",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsKeystorePassword},type=PKCS12)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsTS:add(path="keystore/truststore.p12",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsTruststorePassword},type=PKCS12)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-manager=httpsKM:add(key-store=httpsKS,algorithm="SunX509",credential-reference={store=defaultCS, alias=httpsKeystorePassword})'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/trust-manager=httpsTM:add(key-store=httpsTS)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/server-ssl-context=https:add(key-manager=httpsKM,protocols=["TLSv1.3","TLSv1.2"],use-cipher-suites-order=false,cipher-suite-filter="TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",cipher-suite-names="TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256",trust-manager=httpsTM,want-client-auth=true,authentication-optional=true)'

BASH

Add HTTP(S) Listeners

To add HTTP(S) listeners:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=http:add(socket-binding="http", redirect-socket="https")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=https:add(socket-binding="https", ssl-context="https", max-parameters=2048)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

BASH

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure the Firewall

Open port 8080 and 8443 for incoming TCP traffic.

Open ports in RHEL / CentOS

systemctl enable firewalld --now
firewall-cmd --set-default-zone=dmz
firewall-cmd --zone=dmz --permanent --add-port 8080/tcp
firewall-cmd --zone=dmz --permanent --add-port 8443/tcp
firewall-cmd --reload

CODE

Alternatively, you may open port 80 and port 443 and redirect traffic to 8080 and 8443 respectively.

Set up NAT in RHEL / CentOS

systemctl enable firewalld --now
firewall-cmd --set-default-zone=dmz
firewall-cmd --zone=dmz --permanent --add-port 80/tcp
firewall-cmd --zone=dmz --permanent --add-port 443/tcp
firewall-cmd --permanent --add-forward-port=port=80:proto=tcp:toport=8080
firewall-cmd --permanent --add-forward-port=port=443:proto=tcp:toport=8443
firewall-cmd --reload

CODE

Use an HSM

You can store the private key for the TLS client certificate in an HSM instead for improved security and performance. WildFly contains the Sun PKCS11 provider which can be used to talk to the HSM using PKCS#11.

First, create a Sun PKCS11 configuration file according to the following Jinja2 template. Consult your HSM vendor for details on how to set PKCS#11 attributes.

/opt/wildfly/standalone/configuration/pkcs11.cfg

name = HSM
library = {{ PKCS11_LIBRARY }}
slot = {{ SLOT_NUMBER }}
attributes(*, CKO_PUBLIC_KEY, *) = {
  CKA_TOKEN = false
  CKA_ENCRYPT = false
  CKA_VERIFY = true
  CKA_WRAP = false
}
attributes(*, CKO_PRIVATE_KEY, *) = {
  CKA_TOKEN = true
  CKA_PRIVATE = true
  CKA_SENSITIVE = true
  CKA_EXTRACTABLE = false
  CKA_DECRYPT = false
  CKA_SIGN = true
  CKA_UNWRAP = false
  CKA_DERIVE = false
}

CODE

Make the file readable by WildFly.

chown wildfly:wildfly /opt/wildfly/standalone/configuration/pkcs11.cfg

CODE

Store the token PIN in your credential store.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=hsm,secret-value=123456)'

CODE

Create a provider loader and keystore for the Sun PKCS11 provider.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/provider-loader=pkcs11:add(class-names=[sun.security.pkcs11.SunPKCS11],path=pkcs11.cfg,relative-to=jboss.server.config.dir,module=sun.jdk)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKSInHSM:add(credential-reference={store=defaultCS, alias=hsm},type=PKCS11,providers=pkcs11)'

CODE

Generate a keypair and create a CSR using JBoss CLI. The CSR is stored in /opt/wildfly/standalone/configuration/keystore.

RSA key

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKSInHSM:generate-key-pair(alias=tlsKey0001, distinguished-name="CN=hostname", algorithm=RSA, key-size=2048)' 
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKSInHSM:generate-certificate-signing-request(alias=tlsKey0001, path=keystore/csr.pem, relative-to=jboss.server.config.dir, signature-algorithm=SHA256withRSA)

CODE

Issue the certificate from your favourite CA and put the certificate chain next to the CSR. Then install it on the token:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKSInHSM:import-certificate(alias=tlsKey0001, path=keystore/chain.pem, relative-to=jboss.server.config.dir)'

CODE

Switch to the HSM keystore.

/subsystem=elytron/key-manager=httpsKM:write-attribute(name=key-store,value=httpsKSInHSM)
/subsystem=elytron/key-manager=httpsKM:write-attribute(name=credential-reference,value={store=defaultCS, alias=hsm})

CODE

Configure Web Services WSDL Location Rewrite

In order for the web services to work correctly in cases when the client reads the endpoint address from the Web Services Description Language (WSDL) and uses client certificate or a different port, you need to configure the WSDL web-host rewriting to use the request host.

To configure the WSDL location, run:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=webservices:write-attribute(name=wsdl-host, value=jbossws.undefined.host)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=webservices:write-attribute(name=modify-wsdl-address, value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

CODE

Optional Configuration

The following sections cover optional configurations.

Remove Welcome Content

Removes the hardcoded welcome page in WildFly. This is not needed if Galleon was used.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/location="\/":remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=handler/file=welcome-content:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

BASH

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

You can also remove the actual files and save some disk space with:

rm -rf /opt/wildfly/welcome-content/

CODE

Redirect to Application for Unknown URLs

Known URLs for SignServer starts with /signserver according to the following example:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/rewrite=redirect-to-app:add(redirect=true,target="/signserver/")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=redirect-to-app:add(priority=1,predicate="method(GET) and not path-prefix(/signserver) and not equals({\%{LOCAL_PORT}, 4447})")'

BASH

Enable HTTP Strict Transport Layer Security

The HTTP Strict-Transport-Security response header (HSTS) defined in RFC 6797 tells the browser to only access the server using HTTPS. If you are not serving any resources on the domain over HTTP, you can enable this to improve security. Start off with a small max-age value (such as 60 seconds) and then gradually increase.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/response-header=hsts:add(header-name="Strict-Transport-Security",header-value="max-age=31536000")'

CODE

You should also redirect any HTTP traffic to HTTPS. The example assumes port 80 is used for HTTP, port 443 for HTTPS and that the server can be accessed using the domain name example.com.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/rewrite=http-to-https:add(redirect="true",target="https://example.com:443%U")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=http-to-https:add(predicate="equals(%p,80)")'

CODE

Enable OCSP Revocation Checking

WildFly can check the validity of client certificates against the OCSP responder defined by the certificate's AIA extension:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/trust-manager=httpsTM:write-attribute(name=ocsp, value={})'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

BASH

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Remove the ExampleDS Datasource

You can remove the ExampleDS datasource as it is not being used. This is not needed if Galleon was used.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=ee/service=default-bindings:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect 'data-source remove --name=ExampleDS'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

CODE

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Remove Unused Subsystems and Extensions

Optionally remove the unused subsystems and extensions. This is not needed if Galleon was used.

If you used the Jakarta EE Full & Web Distribution zip package:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=jdr:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=sar:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=jmx:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=pojo:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-metrics-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-jwt-smallrye:remove()' 
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=ee-security:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-health-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-opentracing-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=distributable-web:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=security/security-domain=jaspitest:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/jdbc-driver=h2:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=microprofile-config-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=request-controller:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=security-manager:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.config-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.metrics-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.jwt-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.clustering.web:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.health-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.microprofile.opentracing-smallrye:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.jboss.as.jdr:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.jboss.as.jmx:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.jboss.as.sar:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.jboss.as.pojo:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.ee-security:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.request-controller:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/extension=org.wildfly.extension.security.manager:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

CODE

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Remove AJP

Clean up all AJP related configuration if you are not using it.

/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=ajp:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

CODE

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Enable AJP Connector

Enabling the AJP connector is only needed if you terminate the TLS connection at a proxy (such as Apache or nginx) running in front of WildFly:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/ajp-listener=ajp-listener:add(socket-binding=ajp, scheme=https, enabled=true)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

BASH

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Add a Request Limiter

You can limit the number of concurrent connections by adding a request limiter in Undertow. For example, to allow 100 connections to be processed at the same time, and allow up to 300 connections to be queued before new connections are rejected:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/request-limit=signserver-request-limiter:add(max-concurrent-requests=100,queue-size=300)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=signserver-request-limiter:add(predicate=path-prefix(/signserver)'

CODE

Restrict Access to Services

You can whitelist IP addresses or block access to some services completely using predicates and handlers in Undertow.

For example, to only allow access to the adminstration web from localhost:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/expression-filter=local-only:add(expression="ip-access-control(acl={127.0.0.0/8 allow})")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=local-only:add(predicate="path-prefix(/signserver/adminweb)")'

CODE

To block access to the public web pages and the SignServer documentation:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/configuration=filter/expression-filter=not-found:add(expression="response-code(404)")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/filter-ref=not-found:add(predicate="path-prefix(/signserver/doc) or path(/signserver)")'

CODE

Only Deploy at Startup

Historically application servers are really bad at cleaning up memory from previous deployments and hot (re-)deploy is discouraged in production. To avoid manual deployment with the management interface, we can specify that the deployment directory should be scanned once at application server startup by setting the scan-interval to 0. This also prevents an attacker from loading a malicious JAR file by dropping it in the deployments directory.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=deployment-scanner/scanner=default:write-attribute(name=scan-interval,value=0)'

BASH

Increase the Deployment Timeout

If you are using HSMs with smart card authentication or if the database needs to reindex when WildFly boots, you may have to increase the deployment timeout to be able to deploy SignServer correctly. The deployment timeout is specified in seconds, the command below sets it to 5 minutes.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=deployment-scanner/scanner=default:write-attribute(name=deployment-timeout,value=300)'

BASH

Disable Management Web Console

If you only plan on using the JBoss CLI anyway there is little reason to keep this around.

/opt/wildfly/bin/jboss-cli.sh --connect '/core-service=management/management-interface=http-interface:write-attribute(name=console-enabled,value=false)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

BASH

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Increase the Maximum Upload Size

Note that WildFly defaults to an HTTP post size limit of 10 MB. To allow signing larger files, increase the limits on the HTTP/HTTPS listeners using the max-post-size attribute in the following code examples.

Increase the maximum size of POST requests to for instance 25 MB. This may be needed if you like to sign larger files.

If 3-port separation is used, increase the maximum upload size for the 3 listeners.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=http:write-attribute(name=max-post-size,value=25485760)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=httpspriv:write-attribute(name=max-post-size,value=25485760)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=httpspub:write-attribute(name=max-post-size,value=25485760)' 
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

BASH

If 2-port separation is used, increase the maximum upload size for the 2 listeners instead.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=http:write-attribute(name=max-post-size,value=25485760)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=https:write-attribute(name=max-post-size,value=25485760)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

BASH

Enable WildFly Audit Logging

WildFly can audit log changes made to the management model and output audit log events in JSON format to the file /opt/wildfly/standalone/data/audit-log.log. Audit logging is disabled by default. To enable it, run:

/opt/wildfly/bin/jboss-cli.sh --connect '/core-service=management/access=audit/logger=audit-log:write-attribute(name=enabled,value=true)'

CODE

Remove WildFly Audit Logging

You may remove the WildFly audit logging configuration completely if you are not using it.

/opt/wildfly/bin/jboss-cli.sh --connect '/core-service=management/access=audit:remove()'

BASH

Start WildFly on System Boot

To make SignServer available automatically after a system restart, use systemd.

systemctl enable wildfly

CODE

Performance Tuning

The following section covers WildFly configuration for maximizing SignServer performance. The best setup could vary from installation to installation and should be manually fine-tuned. However, the "default" configuration would typically be a good starting point. You should also consider increasing the heap (RAM) for SignServer instances under high load to at least 4 GB.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=io/worker=default/:write-attribute(name=task-core-threads,value=25)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=io/worker=default/:write-attribute(name=task-max-threads,value=100)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=io/worker=default/:write-attribute(name=io-threads,value=100)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=ejb3/strict-max-bean-instance-pool=slsb-strict-max-pool:undefine-attribute(name=derive-size)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=ejb3/strict-max-bean-instance-pool=slsb-strict-max-pool:write-attribute(name=max-pool-size, value=32)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds/:write-attribute(name=max-pool-size,value=150)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

CODE

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Create a Snapshot of the Configuration

Create a snapshot of the current configuration to make sure you can revert back to a working state.

/opt/wildfly/bin/jboss-cli.sh --connect ':take-snapshot(name="Initial configuration")'
cp /opt/wildfly/standalone/configuration/standalone.xml /opt/wildfly/standalone/configuration/standalone.xml.backup

BASH

Create a Terminal Alias for Reading the Log File

SignServer is logging all actions to the WildFly log file located in /opt/wildfly/standalone/log/server.log. It is often useful to be able to quickly filter and inspect the latest log lines. If you are using Fish you can add the following function to your config.fish:

function wflog --description 'Tail and optionally filter the WildFly log file'
    tail -f /opt/wildfly/standalone/log/server.log | \
        awk '/TRACE/ {print "\033[93m" $0 "\033[39m"}
             /DEBUG/ {print "\033[0;32m" $0 "\033[39m"}
             /INFO/ {print "\033[0;34m" $0 "\033[39m"}
             /WARN/ {print "\033[0;33m" $0 "\033[39m"}
             /ERROR/ {print "\033[0;31m" $0 "\033[39m"}
             /SEVERE/ {print "\033[1;31m" $0 "\033[39m"}
             !/(TRACE|DEBUG|INFO|WARN|ERROR|SEVERE)/ {print "\033[93m" $0 "\033[39m"}' | \
        grep --line-buffered --color=never -E "$argv[1]"
end

BASH

Or if you are using Bash, add this to your .bashrc:

wflog() { 
    tail -f /opt/wildfly/standalone/log/server.log | \
        awk '/TRACE/ {print "\033[93m" $0 "\033[39m"}
             /DEBUG/ {print "\033[0;32m" $0 "\033[39m"}
             /INFO/ {print "\033[0;34m" $0 "\033[39m"}
             /WARN/ {print "\033[0;33m" $0 "\033[39m"}
             /ERROR/ {print "\033[0;31m" $0 "\033[39m"}
             /SEVERE/ {print "\033[1;31m" $0 "\033[39m"}
             !/(TRACE|DEBUG|INFO|WARN|ERROR|SEVERE)/ {print "\033[93m" $0 "\033[39m"}' | \
        grep --line-buffered --color=never -E "$argv[1]"
}

BASH

Clear Command Line History

Clear the command line history to prevent accidental viewing of passwords put into the credential store.

history -c

BASH

References

The following lists links to references and useful external resources.

Next Step: Install SignServer

For instructions on how to install SignServer, see Install SignServer .

Debug SignServer

The following explains the configuration options in WildFly for troubleshooting SignServer.

Switch to Debug Logging Globally

To enable debug logging globally, if you need to troubleshoot, follow these steps.

Make a note of the current log level for the packages org.cesecore and org.signserver.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:read-attribute(name=level)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.signserver:read-attribute(name=level)'

CODE

Switch to debug logging.

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.signserver:write-attribute(name=level, value=DEBUG)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:write-attribute(name=level, value=DEBUG)'

BASH

Once you are done troubelshooting, switch back to the log level you used previously. Note that having debug logging enabled globally will produce a lot of output, avoid using this configuration in production and consider enabling debugging for individual packages instead.

Log Requests and Responses for the WebService API

To log the SOAP messages received by and sent from SignServer:

/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.cxf.logging.enabled:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf.services:add(level=INFO)'
systemctl restart wildfly

CODE

To remove the configuration when you are done:

/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.cxf.logging.enabled:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf.services:remove'
systemctl restart wildfly

CODE

Enable Hibernate Statistics

Hibernate is the ORM library used by SignServer to create SQL queries. You can enable Hibernate statistics to get the SQL query, as well as the time it takes to execute it, written in the server log.

Hibernate statistics are quite verbose and there may be database-specific tools better suited to troubleshoot database performance, however if you want to enable it:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds/statistics=pool:write-attribute(name=statistics-enabled,value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds/statistics=jdbc:write-attribute(name=statistics-enabled,value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.generate_statistics:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.show_sql:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.format_sql:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.use_sql_comments:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate.stat:add(level=DEBUG)'
systemctl restart wildfly

CODE

To remove the configuration:

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate.stat:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.use_sql_comments:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.format_sql:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.show_sql:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.generate_statistics:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds/statistics=jdbc:write-attribute(name=statistics-enabled,value=false)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds/statistics=pool:write-attribute(name=statistics-enabled,value=false)'
systemctl restart wildfly

CODE

Enable Profiling Using Glowroot

Download Glowroot and put it in your WildFly directory.

wget https://github.com/glowroot/glowroot/releases/download/v0.13.6/glowroot-0.13.6-dist.zip -O /tmp/glowroot.zip
unzip -q /tmp/glowroot.zip -d /opt/wildfly
chown -R wildfly:wildfly /opt/wildfly/glowroot

CODE

Register Glowroot as a Java agent and restart WildFly.

sed -i '/-Djdk.tls.ephemeralDHKeySize=2048/ a \ \ \ JAVA_OPTS=\"$JAVA_OPTS -javaagent:/opt/wildfly/glowroot/glowroot.jar"' /opt/wildfly/bin/standalone.conf
systemctl restart wildfly

CODE

Glowroot should now be available on http://localhost:4000.