Enabling Key Trustee KMS High Availability

CDH 5.4.0 and higher supports Key Trustee KMS high availability. For new installations, you can use the Set up HDFS Data At Rest Encryption wizard to install and configure Key Trustee KMS high availability. If you have an existing standalone Key Trustee KMS service, use the following procedure to enable Key Trustee KMS high availability:

Back up the Key Trustee KMS private key and configuration directory. See Backing Up and Restoring Key Trustee Server and Clients for more information.
If you do not have a ZooKeeper service in your cluster, add one using the instructions in Adding a Service.
Run the Add Role Instances wizard for the Key Trustee KMS service (Key Trustee KMS service > Actions > Add Role Instances).
Click Select hosts and check the box for the host where you want to add the additional Key Management Server Proxy role. See Resource Planning for Data at Rest Encryption for considerations when selecting a host. Click OK and then Continue.
On the Review Changes page of the wizard, confirm the authorization code, organization name, and settings, and then click Finish.
Important: The initial startup of the KMS instance may fail with the following error message:
```
java.io.IOException: Unable to verify private key match between KMS hosts. If the system has been recently upgraded, DO NOT TAKE FURTHER ACTION and contact your support representative as soon as possible. If this is a new installation, verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.
```
If this occurs, it indicates that the KMS attempted to verify the Key Trustee private key has been synchronized with the new instance, but was unable to because that synchronization has not yet taken place. This is expected behavior at this point in the process. Proceed to the next step, and the new KMS instance will come up when the KMS service is restarted after the synchronization.
If it is not already running, start the new KMS instance. Select the new instance and go to Actions for Selected > Start.
Go to Key Trustee KMS service > Configuration and make sure that the ZooKeeper Service dependency is set to the ZooKeeper service for your cluster.
Synchronize the Key Trustee KMS private key.
Warning: It is very important that you perform this step. Failure to do so leaves Key Trustee KMS in a state where keys are intermittently inaccessible, depending on which Key Trustee KMS host a client interacts with, because cryptographic key material encrypted by one Key Trustee KMS host cannot be decrypted by another. If you are already running multiple Key Trustee KMS hosts with different private keys, immediately back up all Key Trustee KMS hosts, and contact Cloudera Support for assistance correcting the issue.
If you fail to maintain proper synchronization of private keys between Key Trustee KMS hosts, then the GPG validation check that runs automatically when the Key Trustee KMS is restarted will return the following error and abort the restart operation, forcing you to synchronize private keys before a restart can occur:
```
java.io.IOException: Unable to verify private key match between KMS hosts. Verify private key files have been synced
between all KMS hosts. Aborting to prevent data inconsistency.
```
To determine whether the Key Trustee KMS private keys are different, compare the MD5 hash of the private keys. On each Key Trustee KMS host, run the following command:
```
$ md5sum /var/lib/kms-keytrustee/keytrustee/.keytrustee/secring.gpg
```
If the outputs are different, contact Cloudera Support for assistance. Do not attempt to synchronize existing keys. If you overwrite the private key and do not have a backup, any keys encrypted by that private key are permanently inaccessible, and any data encrypted by those keys is permanently irretrievable. If you are configuring Key Trustee KMS high availability for the first time, continue synchronizing the private keys.
Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network by running the following rsync command on the original Key Trustee KMS host:
```
rsync -zav /var/lib/kms-keytrustee/keytrustee/.keytrustee root@ktkms02.example.com:/var/lib/kms-keytrustee/keytrustee/.
```
Replace ktkms02.example.com with the hostname of the Key Trustee KMS host that you are adding.
Restart the Key Trustee KMS service (Key Trustee KMS service > Actions > Restart).
Restart the cluster.
Redeploy the client configuration (Home > Cluster-wide > Deploy Client Configuration).
Re-run the steps in Validating Hadoop Key Operations.

Cloudera Navigator Key Trustee Server High Availability

Enabling Navigator HSM KMS High Availability