Backing up Cloudera Data Engineering service using the docker image

You must run the docker image to take backup of a Cloudera Data Engineering service. It takes backup of all the active virtual clusters in that Cloudera Data Engineering service. You can take backup of only an active Cloudera Data Engineering service.

You must download the dex-upgrade-utils docker image and create the cde-upgrade-util.properties file before backing up jobs as described in the Prerequisites for upgrading Cloudera Data Engineering Service with endpoint stability section.

Run the dex-upgrade-utils docker image on the host machine:

$ export BACKUP_OUTPUT_DIR=/home/dex/backup 

docker run \
-v [***KUBECONFIG_FILE_PATH***]:/home/dex/.kube/config:ro \
-v [***CDP_CREDENTIAL_FILE_PATH***]:/home/dex/.cdp/credentials:ro \
-v [***CDE-UPGRADE-UTIL.PROPERTIES_FILE_PATH***]:/opt/cde-backup-restore/scripts/backup-restore/cde-upgrade-util.properties:ro \
-v [***LOCAL_BACKUP_DIRECTORY***]:$BACKUP_OUTPUT_DIR \
-e KUBECONFIG=/home/dex/.kube/config \
[***DOCKER_IMAGE_NAME***]:[***DOCKER_IMAGE_VERSION***] prepare-for-upgrade -s [***CDE-CLUSTER-ID***] -o $BACKUP_OUTPUT_DIR

Example:

$ docker run \

-v $BASE_WORK_DIR/secrets/kubeconfig:/home/dex/.kube/config:ro \
-v $BASE_WORK_DIR/secrets/credentials:/home/dex/.cdp/credentials:ro \
-v $BASE_WORK_DIR/cde-upgrade-util.properties:/opt/cde-backup-restore/scripts/backup-restore/cde-upgrade-util.properties:ro \
-v $BASE_WORK_DIR/backup:$BACKUP_OUTPUT_DIR \
-e KUBECONFIG=/home/dex/.kube/config \
docker-private.infra.cloudera.com/cloudera/dex/dex-upgrade-utils:1.20.1-b48 prepare-for-upgrade -s cluster-c2dhkp22 -o $BACKUP_OUTPUT_DIR

To identify the CDE-CLUSTER-ID, do the following steps:

In the Cloudera console, click the Data Engineering tile. The Cloudera Data Engineering home page displays.
On the left navigation menu, click Administration.
In the Services column, click for the Service for which you want to identify the Cluster ID.
Note the CLUSTER ID displayed on the page.

You have now taken the Cloudera Data Engineering service backup as a ZIP file. You can make a note of the Zip file name from the logs to use it while restoring the Cloudera Data Engineering service.

You must now expand the resource pool, and then upgrade your Cloudera Platform before you restore the Cloudera Data Engineering service. For information about configuring resource pool and capacity, see Managing cluster resources using Quota Management.

important

During the restore operation, both old and the new Cloudera Data Engineering services use the same resources allocated to the existing Cloudera Data Engineering service. Hence, you must double the resource pool size using the Quota Management option. For example, if root.default.sales is the pool that is used for the old or existing Cloudera Data Engineering service, you must double the CPU and memory resources of this pool. Also, make sure that you have sufficient hardware when doubling the resource pool size. Consider the following conditions and plan whether to modify the resource pool size or not:

If the Cloudera Data Engineering service uses the default resource pool, that is root.default, then do not change the resource pool size.
If the Cloudera Data Engineering service uses a custom resource pool (for example, root.default.primary.secondary), the resource pool size of the last level (that is., secondary level in the example) must be doubled using Quota Management option. The additional capacity required after doubling the last level's pool size is allocated from the levels above it, starting from the higher levels and progressing downward. In this example, when you double the secondary level (last level), the extra resource pool capacity required is initially added to the primary level pool. Then the newly added resource pool capacity is added to the secondary level pool, resulting in an overall doubling of the resource pool size of the last level.
The resource capacity at the Cloudera Data Engineering service and the Virtual Cluster level must not be changed. Modifying the pool size at the resource pool level is sufficient.