Key Trustee KMS Encryption Issues

Description: The following error occurs when attempting to restart the Key Trustee KMS after a host failure on a cluster:

java.io.IOException: Unable to verify private key match between KMS hosts. Verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.        

Solution: If you have failed to synchronize private keys between Key Trustee KMS hosts, they may be in a state where keys are intermittently inaccessible, depending on which Key Trustee KMS host a client interacts with, because cryptographic key material encrypted by one Key Trustee KMS host cannot be decrypted by another. If you are already running multiple Key Trustee KMS hosts with different private keys, immediately back up all Key Trustee KMS hosts, and contact Cloudera Support for assistance correcting the issue.

Description: You may see the following error after you attempt to restart a KT KMS HA host after an upgrade:

java.io.IOException: Unable to verify private key match between KMS hosts. Verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.         

Solution: See Validating Private Key Synchronization (Key Trustee KMS HA Only) for guidance on synchronization and validation of private keys.

Description: You may see the following error after you attempt to restart a Key Trustee KMS for the first time:

java.lang.Exception: Could not establish connection to ZooKeeper to verify KMS host private key consistency. Verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.        

Solution: ZooKeeper is used to communicate with hosts and is also the storage location of private key data, and therefore must be running upon the first restart or running of the GPG validation check, which compares private keys amongst Key Trustee KMS hosts to help prevent a "split brain" scenario (when private keys are not synchronized between hosts). To ensure the GPG validation check can run, start ZooKeeper, and then restart the Key Trustee KMS.