Typically, when using SignServer and creating CryptoTokens, a CryptoToken will generally look at a slot in an HSM for its key material. You would then create multiple crypto workers to look in different HSM slots.  

When using CloudHSM, the CloudHSM client provided by AWS always looks at a single cluster of HSMs. This cluster is always referenced as slot 1 to the host accessing it. AWS' CloudHSM client does not allow a host to access more than one CloudHSM cluster at a time. SignServer supports more than one crypto worker that can look at different HSMs, but because of the CloudHSM clients ability to only talk to a single HSM cluster and represent it as slot 1, SignServer ECE can only talk to a single CloudHSM cluster at a time. On-prem HSM solutions can still work in the typical manner that the HSM provider supports.

The only limitation that this presents is that all of the signing keys for SignServer workers must be located in the same slot of the CloudHSM.  You can have more than one key in the HSM and call that by different signing workers.