Maintenance
Events like installation of updates, or raid failures can impede the operation of Hardware Appliance services. In such cases, you will not see long error messages. Instead, the Hardware Appliance will be put into maintenance state.
In maintenance state, all access to EJBCA/SignServer over HTTP(S) is disabled. Each request will return a message informing you that the system is in maintenance and cannot be accessed.
Hardware Appliance States
To find out the state of the Hardware Appliance, go to the page Platform > Troubleshooting in WebConf. The Hardware Appliance can be in one of three different states:
- Operational
The Hardware Appliance is fully operational. All subsystems are working as expected. - Maintenance
The Hardware Appliance is in maintenance mode and application services are interrupted by an automatically detected reason.
The state Maintenance can replace the state Offline. - Offline
The Hardware Appliance is in maintenance mode and application services are interrupted by a manual setting in WebConf.
- The Hardware Appliance can only switch to Offline status if this is set manually. Every automatically detected reason changes the status from Offline to Maintenance. In such a case, the manual Offline setting is still active but invisible. If all automatically detected reasons are removed, the Hardware Appliance switches back to Offline status.
- Most customers will not see any difference between the Offline and Maintenance state. Operators, however, will know that the Offline state indicates a maintenance mode that is manually set in WebConf. In contrast, the automatic Maintenance state indicates a real problem. Here, the Hardware Appliance services were not taken offline by choice.
- It makes sense during an automatically induced Maintenance state to also set the Hardware Appliance manually Offline in WebConf: Operators can then check the integrity of the Hardware Appliance after an incident before exposing services to customers.
Reasons for Maintenance state
The Hardware Appliance will be put into Maintenance state automatically for the following reasons:
- Factory Reset During Operation
The Hardware Appliance will be set to Maintenance automatically if the Factory Reset procedure is triggered during operation. The Maintenance state ends with the next reboot finishing the Factory Reset.
This event is not recoverable! - RAID Failure
The Hardware Appliance will be set to Maintenance automatically if a fatally broken RAID is detected. The reason for this:
The Hardware Appliance will enter an inconsistent state if both SSD hard disk drives fail. This state cannot trigger any error messages until caches are finally flushed. Setting Maintenance with one broken RAID ensures that no data is created that cannot be recovered later.
This event is not recoverable! - HSM Alarm
The Hardware Appliance will be set to Maintenance automatically if the embedded HSM has detected an alarm. Due to the alarm, all key materials will be erased by the HSM. The operation of EJBCA/SignServer without a functioning HSM is therefore not reasonable.
This event is not recoverable! - Database is Down
The Hardware Appliance will be set to Maintenance if the embedded database system stops operating. When the database is available again Maintenance state is stopped automatically.
This event is recoverable! - Application Update
The Hardware Appliance will be set to Maintenance if an application is updated via WebConf. When the update has finished Maintenance state is stopped automatically.
This event is recoverable! Manual Setting 'Offline'
You can use the Offline function in the WebConf page Platform > Troubleshooting to manually activate the maintenance mode for the Hardware Appliance. In contrast to the automatic Maintenance state, the Offline state is not based on any apparent problem/cause. You can use this function to disable customer access to EJBCA/SignServer without completely shutting down the Hardware Appliance. Customers will then see the Notification page that is described below.
The Offline state will not persist a reboot of the Hardware Appliance.
Effects of the Maintenance state
The following sections describe changes and information shown when the Hardware Appliance is operating in maintenance.
Notification Page
Every HTTP(S) request to EJBCA/SignServer will lead to an HTTP 501 status code response. A web page appears and notifies the user that the Hardware Appliance is currently not operational and running in maintenance.
OCSP requests will also receive an HTTP 501 status code with that notification page inside the responses body.
Front Display
When the Hardware Appliance enters maintenance the messages on the front display include the following:
MAINTENANCE
Services unavail
The message disappears when the state of the Hardware Appliance switches back to operational.
WebConf
Troubleshooting Section
In the WebConf page Platform > Troubleshooting all maintenance reasons will be listed.
If the Hardware Appliance is set to Offline this will only be reflected by a change in the button: the button name Offline switches to Online.
Warning Messages
When a WebConf page is opened during maintenance the white-on-red message Services Unavailable appears in the upper left corner. After leaving maintenance, the message will disappear when the page is reloaded or when a new page is opened.
SNMP
If SNMP is enabled it will indicate the Hardware Appliance state. it will also show a human-readable combined message of all reasons for maintenance. For more details refer to section Monitoring > SNMP.
Syslog
Syslog and AVM server log will contain detailed messages about changing events that lead to state changes of the Hardware Appliance.
Support Package
Each time the Hardware Appliance enters maintenance a Support Package will be created. This also happens if the Hardware Appliance has been set Offline manually.
If the Hardware Appliance is already in maintenance state, no additional Support Package will be created. For example, if the SSD harddisk drives all fail and the Hardware Appliance is automatically set to the Maintenance state. Minutes later the Factory Reset is triggered. In such a case, only one Support Package will be created. For more information, see Support Packages.
Required Maintenance Procedure on the Software
Yearly Maintenance Task
We strongly recommend updating the software at least once a year!
For further information, see the Upgrade Notes and Hardware Appliance Release Information!
Required Maintenance Procedure on the External Battery Adapter
The external battery adapter has the task to support the power supply of the HSM. The HSM always requires power, even when the Hardware Appliance has not been put into operation or is not supplied with power. To prevent the HSM from being fully discharged and thus placed in maintenance mode, use the external battery adapter to buffer the internal HSM battery and extend its lifetime when the Hardware Appliance is powered off. This is useful, for example, when a Hardware Appliance is operated as an offline (root) CA.
Quarterly Maintenance Task
Monitor the state of the battery via WebConf:
In the WebConf page Status > Overview page the state of the HSM battery and the external battery can be observed.
Make sure that the battery of the external battery adapter is in a good state of charge. Otherwise, please replace it immediately.
Yearly Maintenance Task
At an annual interval, be sure to replace the battery of the external battery adapter on the back of the device. This is a safety measure that should be observed!
- Turn to the back of the device.
- Carefully unplug the external battery adapter from its connector.
External Battery Adapter mounted on Hardware Version 2
External Battery Adapter mounted on Hardware Version 4 - Remove the battery from its connection.
- Replace the battery with a new one (standard 6LR61/9 Volt block battery) and reconnect the battery and the adapter to the device. Please observe the environmental and resource protection regulations when disposing of the used battery.