June 25, 2025
This release (1.24.1) of the Cloudera Data Engineering service on Cloudera on cloud introduces the following changes.
Suspending and resuming Cloudera Data Engineering Services - Technical preview
The suspend and resume feature enables you to temporarily pause your entire Cloudera Data Engineering Service during idle periods. Suspending Services when they are not needed offers significant cost savings, primarily on compute resources, while ensuring that the Services remain available when required. During suspension, all Service and Virtual Cluster configurations and Jobs are preserved. By default, the entitlement for the suspend and resume feature is disabled. For more information, see Suspending and resuming Cloudera Data Engineering Services.
Security hardened image
Starting with Cloudera Data Engineering version 1.24.1, a new security-hardened image is introduced to enhance security and increase resilience against Common Vulnerabilities and Exposures (CVEs). In Cloudera Data Engineering version 1.24.1, you have the option to choose between adopting the new security hardened image or continuing to use the existing image from previous versions (Redhat UBI), based on your system needs and business requirements.
The runtime component versions are upgraded in the security hardened image; for compatibility, see Compatibility for Cloudera Data Engineering and Runtime components. For information on the upgrade behavior of the security hardened image, see Cloudera Data Engineering upgrade version compatibility.
Git integration support available for GA
The Git integration support feature, previously available as a technical preview, has now reached General Availability (GA). For more information, see Creating a Git repository in Cloudera Data Engineering.
Autoscaling range setting method enhanced on the Cloudera Data Engineering UI
While creating a new Cloudera Data Engineering Service on the Cloudera Data Engineering UI (), you have the option to set the autoscaling range for the Core and All-Purpose Tiers. In the Capacity & Costs section, you can configure the autoscaling range for On-demand instances and if you select an AWS environment, for Spot instances as well. For more information, see Cloudera Data Engineering tier types.
Runtime component version exposure on the Cloudera Data Engineering UI
On the Cloudera Data Engineering UI, you can view the Apache Airflow runtime components and Spark runtime components on the creation and details pages of a Service, Virtual Cluster, job, or session.
The following screenshot illustrates creating a new Virtual Cluster, where you can select the required runtime component versions from a drop-down menu on the UI.
Runtime component version behavior after upgrading
To maintain backwards compatibility, the in-place upgrade does not modify the image type, and after the upgrade to Cloudera Data Engineering version 1.24.1, the old RedHat (deprecated) images continue to be used. To use the new, security hardened image, after the upgrade, you must create new Virtual Clusters and select the new image. For more information on the supported runtime components, see Compatibility for Cloudera Data Engineering and Runtime components.
Cloudera Data Engineering private network for Elastic Kubernetes Service and Azure Kubernetes Service available for GA
Kubernetes version upgrade to 1.31
The Kubernetes version that Cloudera Data Engineering uses is upgraded to Kubernetes 1.31. For more information, see Compatibility for Cloudera Data Engineering and Runtime components.
Apache Airflow version upgrade to 2.10.4
The Airflow version that Cloudera Data Engineering uses is upgraded to Airflow 2.10.4. For more information, see:
Fixed issues
- DEX-15798: [Postgres][Azure] Private DNS zone is not created
- DEX-15989: Diagnostics Bundle lacks App Instances information
- DEX-16007: Spark 3.3+ on JDK 17 errors
- DEX-17281: Unmounted Airflow file mount still present at job runtime
- DEX-17264: [BnR] vc.vcApiUrl is not present in the VC Describe
- DEX-17495: In AKS 1.30+ Spark2 jobs running more than 1 hour are failing
