Backing Up and Restoring Cloudera Search

Before you begin backing up Cloudera Search, make sure that solr.xml contains following configuration section:

  <backup>
    <repository name="hdfs"     class="org.apache.solr.core.backup.repository.HdfsBackupRepository" default="true">
      <str name="solr.hdfs.home">${solr.hdfs.home:}</str>
      <str name="solr.hdfs.confdir">${solr.hdfs.confdir:}</str>
      <str name="solr.hdfs.security.kerberos.enabled">${solr.hdfs.security.kerberos.enabled:false}</str>
      <str name="solr.hdfs.security.kerberos.keytabfile">${solr.hdfs.security.kerberos.keytabfile:}</str>
      <str name="solr.hdfs.security.kerberos.principal">${solr.hdfs.security.kerberos.principal:}</str>
      <str name="solr.hdfs.permissions.umask-mode">${solr.hdfs.permissions.umask-mode:000}</str>
    </repository>
  </backup>

New installations of CDH 5.9 and higher include this section. For upgrades, you must manually add it and restart the Solr service.

To edit the solr.xml file:

1. Download solr.xml from ZooKeeper:

   $ solrctl cluster --get-solrxml $HOME/solr.xml

2. Edit the file locally and add the <backup> section at the end of the solr.xml file before the closing </solr> tag:

   $ vi $HOME/solr.xml

3. Upload the modified file to ZooKeeper:

   $ solrctl cluster --put-solrxml $HOME/solr.xml

As part of backup/restore mechanism, the following APIs are introduced. The following table identifies the Sentry authorization permissions to configure. For information on configuring Sentry privileges, see Configuring Sentry Authorization for Cloudera Search.

Use the following procedure to back up a Solr collection. For more information on the commands used, see Cloudera Search Backup and Restore Command Reference.

If you are using a secure (Kerberos-enabled) cluster, specify your jaas.conf file by adding the following parameter to each command:

--jaas /path/to/jaas.conf

If TLS is enabled for the Solr service, specify the truststore and password using the ZKCLI_JVM_FLAGS environment variable before you begin the procedure:

$ export ZKCLI_JVM_FLAGS="-Djavax.net.ssl.trustStore=/path/to/truststore \
-Djavax.net.ssl.trustStorePassword=trustStorePassword"

1. Create a snapshot. On a host running Solr Server, run the following command:

   $ solrctl collection --create-snapshot <snapshotName> -c <collectionName>

   For example, to create a snapshot for a collection named tweets:

   $ solrctl collection --create-snapshot tweets-$(date +%Y%m%d%H%M) -c tweets
   Successfully created snapshot with name tweets-201610191429 for collection tweets

2. If you are backing up the Solr collection to a remote cluster, prepare the snapshot for export. If you are backing up the Solr collection to the local cluster, skip this step.

   $ solrctl collection --prepare-snapshot-export <snapshotName> -c <collectionName> -d <destDir>

   The destination HDFS directory path (specified by the -d option) must exist on the local cluster before you run this command. Make sure that the Solr superuser (solr by default) has permission to write to this directory.

   For example:

   $ hdfs dfs -mkdir -p /path/to/backup-staging/tweets-201610191429
   $ hdfs dfs -chown :solr /path/to/backup-staging/tweets-201610191429
   $ solrctl collection --prepare-snapshot-export tweets-201610191429 -c tweets \
   -d /path/to/backup-staging/tweets-201610191429

3. Export the snapshot. This step uses the DistCp utility to back up the collection metadata as well as the corresponding index files. The destination directory must exist and be writable by the Solr superuser (solr by default). To export the snapshot to a remote cluster, run the following command:

   $ solrctl collection --export-snapshot <snapshotName> -s <sourceDir> -d <protocol>://<namenode>:<port>/<destDir>

   For example:

   • HDFS protocol:

     $ solrctl collection --export-snapshot tweets-201610191429 -s /path/to/backup-staging/tweets-201610191429 \
     -d hdfs://nn01.example.com:8020/path/to/backups

   • WebHDFS protocol:

     $ solrctl collection --export-snapshot tweets-201610191429 -s /path/to/backup-staging/tweets-201610191429 \
     -d webhdfs://nn01.example.com:20101/path/to/backups

   To export the snapshot to the local cluster, run the following command:

   $ solrctl collection --export-snapshot <snapshotName> -c <collectionName> -d <destDir>

   For example:

   $ solrctl collection --export-snapshot tweets-201610191429 -c tweets -d /path/to/backups/

4. Delete the snapshot:

   $ solrctl collection --delete-snapshot <snapshotName> -c <collectionName>

   For example:

   $ solrctl collection --delete-snapshot tweets-201610191429 -c tweets

Use the following procedure to restore a Solr collection. For more information on the commands used, see Cloudera Search Backup and Restore Command Reference.

If you are using a secure (Kerberos-enabled) cluster, specify your jaas.conf file by adding the following parameter to each command:

-jaas /path/to/jaas.conf

If TLS is enabled for the Solr service, specify the truststore and password by using the ZKCLI_JVM_FLAGS environment variable before you begin the procedure:

$ export ZKCLI_JVM_FLAGS="-Djavax.net.ssl.trustStore=/path/to/truststore \
-Djavax.net.ssl.trustStorePassword=trustStorePassword"

1. If you are restoring from a backup stored on a remote cluster, copy the backup from the remote cluster to the local cluster. If you are restoring from a local backup, skip this step.

   Run the following commands on the cluster to which you want to restore the collection:

   $ hdfs dfs -mkdir -p /path/to/restore-staging
   $ hadoop distcp <protocol>://<namenode>:<port>/path/to/backup /path/to/restore-staging

   For example:

   • HDFS protocol:

     $ hadoop distcp hdfs://nn01.example.com:8020/path/to/backups/tweets-201610191429 /path/to/restore-staging

   • WebHDFS protocol:

     $ hadoop distcp webhdfs://nn01.example.com:20101/path/to/backups/tweets-201610191429 /path/to/restore-staging

2. Start the restore procedure. Run the following command:

   $ solrctl collection --restore <restoreCollectionName> -l <backupLocation> -b <snapshotName> -i <requestId>

   Make sure that you use a unique <requestID> each time you run this command. For example:

   $ solrctl collection --restore tweets -l /path/to/restore-staging -b tweets-201610191429 -i restore-tweets

3. Monitor the status of the restore operation. Run the following command periodically:

   $ solrctl collection --request-status <requestId>

   Look for <str name="state"> in the output. For example:

   $ solrctl collection --request-status restore-tweets
    <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status"> 0</int> <int name="QTime"> 1</int> </lst> \
   <lst name="status"> <str name="state"> completed</str> <str name="msg"> found restore-tweets in completed tasks</str> </lst> </response>

   The state parameter can be one of the following:

   • running: The restore operation is running.
   • completed: The restore operation is complete.
   • failed: The restore operation failed.
   • notfound: The specified <requestID> does not exist.

-jaas /path/to/jaas.conf

If TLS is enabled for the Solr service, specify the truststore and password by using the ZKCLI_JVM_FLAGS environment variable:

$ export ZKCLI_JVM_FLAGS="-Djavax.net.ssl.trustStore=/path/to/truststore \
-Djavax.net.ssl.trustStorePassword=trustStorePassword"