Fixed Issues

Review the fixed issues in this release of the Cloudera Data Warehouse service on cloud

DWX-17703: Non-HA Impala Virtual Warehouse on a private Azure Kubernetes Service (AKS) setup fails

When 'Refresh' and 'Stop' operations run in parallel, Impala might move into an error state. The Refresh operation might think that Impala is in an error state as the coordinator pod is missing.

The issue is no longer noticed because of leasing. While an object holds a lease during an operation, no other operation can access the object. This makes the 'stop' operation exclusive.

DWX-19451: Cloudera Data Visualization restore job can fail with ignorable errors

After a successful Cloudera Data Visualization restoration job, the restore job could be in a failed state with the log displaying ignorable errors.

This issue occurs because the restore job issues commands to DROP all the objects that will be restored, and if any of these objects do not exist in the destination database, such ignorable errors are reported.

This fix addresses the issue by splitting the backup and restore process into two steps — SCHEMA + DATA and the errored-out container does not fail anymore.

DWX-19595: Database Catalog goes to an error state after a Data Lake resize

After a Cloudera Data Lake resize, the Cloudera Data Warehouse Database Catalog needs to be restarted to consider the change in Data Lake configuration. You can restart the Database Catalog using the CDP CLI or by using the Stop and Start functionality from the Cloudera Data Warehouse UI. However, irrespective of how the Database Catalog is restarted, it ends up in an "Error" state.

This issue is fixed and Cloudera Data Warehouse now recognizes the Data Lake configuration changes regardless of whether the Database Catalog was started through the CDP CLI or through the Cloudera Data Warehouse UI.

DWX-20309: Database Catalog fails to start after upgrading Data Lake to Azure Flexible Server

After you upgrade your Data Lake from Azure Single Server to Azure Flexible Server, the Cloudera Data Warehouse Database Catalog fails to start.

You may restart the Database Catalog using the CDP CLI or by using the Stop and Start functionality from the Cloudera Data Warehouse UI. Irrespective of how the Database Catalog is restarted, it ends up in an "Error" state

This fix addresses this issue and the Cloudera Data Warehouse Database Catalog starts up fine after upgrading the Data Lake from Azure Single Server to Azure Flexible Server.

CDPD-43946: Iceberg tables have a large memory footprint in the catalog cache

Iceberg tables consume significantly more Java Virtual Machine (JVM) heap memory compared to Hive tables having the same structure, such as number of partitions, files, and so on.

This fix addresses the issue and the JVM memory footprint of the Iceberg tables is now similar to Hive tables that have the same structure.

Apache Jira: IMPALA-11265

CDPD-70956: Queries over JDBC tables fail due to column types mismatch

Queries over JDBC tables fail at runtime when there is a mismatch between the Hive type and the database type for some columns and CBO is not used.

Apache jira: HIVE-28285

DWX-17619: HPL/SQL built-in function unexpected output

Certain HPL/SQL built-in functions, such as lower and trim, were not functioning correctly when used in INSERT statements. This issue occurred after the HIVE-27492 fix, which removed UDFs required for HPL/SQL's local and offline modes.

The issue was resolved by re-adding the necessary UDFs to HPL/SQL to ensure compatibility with local and offline modes. Related issues with these UDFs were also fixed to restore their functionality in INSERT and other SELECT statements.

Apache Jira: HIVE-28143

CDPD-74205: SharedWorkOptimizer leaves residual unused operator tree

An issue was identified where the shared work optimizer left behind unused operator trees that sent dynamic partition pruning (DPP) events to non-existing table scan operators. This caused errors during query execution, such as

No work found for
      tablescan TS[53],

disrupting workflows and query processing.

The issue was fixed by removing any leftover operator trees that sent dynamic partition pruning events to unknown operators during the optimization process. The fix ensures smoother query execution and prevents such errors.

Apache Jira: HIVE-28484

CDPD-73269: RexLiteral to ExprNode conversion issue with empty string

The conversion from RexLiteral to ExprNode failed when the literal was an empty string. This issue, introduced in HIVE-23892 caused the Cost-Based Optimizer (CBO) to fail for queries containing filters with empty literals.

The issue was fixed by ensuring that an empty literal in the filter still produces a valid RexNode during the conversion process. This fix prevents CBO failures for such queries.

Apache Jira:HIVE-28431

CDPD-44551: Avro table import or download fails with ODBC driver due to missing property

The absence of metastore.storage.schema.reader.impl caused Avro table import or download failures in Cloudera 7.1.7 when using the ODBC driver.

The issue was addressed by ensuring that all records are correctly preserved during major compaction.

Apache Jira: HIVE-26952

CDPD-72605: Optimized partition authorization in HiveMetaStore to reduce overhead

The add_partitions() API in HiveMetaStore was authorizing both new and existing partitions, leading to unnecessary processing and increased load on the authorization service.

The issue was addressed by modifying the add_partitions() API to authorize only new partitions, improving performance and reducing authorization overhead.

Apache Jira: HIVE-28371

CDPD-73046: Removal of duplicated proto reader/writer classes

Duplicate Java files for proto reader/writer classes were present in Hive, which were already available in Apache Tez. These duplicates caused redundancy and missed improvements introduced in Tez, such as those from TEZ-4296, TEZ-4105, and TEZ-4305

The issue was fixed by removing the duplicated proto reader/writer classes from Hive, ensuring the use of the improved versions available in Apache Tez.

Apache Jira: HIVE-28028

CDPD-56130: Event Processor failure due to table lock release error

The event processor entered an error state when a table write lock attempt timed out. Since the lock was not held by the current thread, an attempt to release it triggered an IllegalMonitorStateException.

The fix added a check to ensure the table holds the write lock before releasing it, preventing event processor failures and unnecessary table invalidations.

Apache Jira: IMPALA-12141

CDPD-57725: Security Enhancement: CSP Nonces Enforced in Hue

Hue now enforces Content Security Policy (CSP) nonces, eliminating unsafe-inline JavaScript to improve security and mitigate InfoSec scan issues related to wildcard domains and other directives.

Update the following in the hue.ini configuration:

[desktop]  
csp_nonce=true

Note that, the Workflow Editor will not be available in this mode.

This update enhances security by reducing the risk of cross-site scripting (XSS) attacks.

CDPD-70407: Improved memory efficiency in IcebergDeleteNode with RoaringBitmap

IcebergDeleteNode uses an ordered 64-bit integer array to store deleted positions, leading to high memory consumption, especially with large delete records. For example, 100 million delete records required 800 MiB of memory.

The fix includes replacing the sorted 64-bit integer array with RoaringBitmap, reducing memory usage while maintaining performance.

Apache Jira: IMPALA-13109

CDPD-79583: Workload management insert DML timeouts

Workload management insert DMLs into sys.impala_query_log were canceled after waiting 10 seconds in admission control, and they did not respect the settings that limited the maximum amount of time an insert DML could execute.

Query options for workload management insert DMLs were modified to prevent premature cancellation. A new flag, query_log_dml_exec_timeout_s, was added to ensure these queries time out properly.

Apache Jira: IMPALA-13772

CDPD-81027: Workload management insert fails due to statement expression limit

Workload management insert operations into sys.impala_query_log failed with an

Exceeded the statement expression
      limit (1024)

error. The issue occurred because the system miscalculated the required statement expression limit, causing queries to exceed the limit.

This issue has been fixed by taking the default statement expression limit and reducing the maximum queued queries limit from 5,000 to 3,000 to prevent large inserts from exceeding the default threshold.

Apache Jira: IMPALA-13881

CDPD-70945: ConcurrentModificationException in ReloadEvent processing

The event processor encountered a ConcurrentModificationException when handling partition-level RELOAD events. This occurred because ReloadEvent.isOlderEvent() checked the partition reload status without holding a table read lock, leading to conflicts with concurrent DDL/DML operations modifying the partition list.

The issue was addressed by ensuring that the event processor acquires a table read lock before checking partition reload status, preventing concurrent modifications from causing errors.

Apache Jira: IMPALA-13126

CDPD-72058: Optimized serialization of position delete records

The serialization of position delete records was inefficient, storing file paths redundantly and increasing memory usage. The process involved unnecessary copying, leading to larger buffers that required compression before transmission.

The fix includes optimizing serialization by grouping position delete records with the same file path.

Apache Jira: IMPALA-13194

CDPD-74539: Maria DB falls back to MySQL in Hive

Hive downstream had errors in supporting Maria DB.

The issue was addressed by making Maria DB automatically fall back to MySQL

CDPD-74683: Incorrect skipping of file metadata reload for ALTER_TABLE events

The optimization introduced in IMPALA-12487 skipped file metadata reload for ALTER_TABLE events if changes in storage descriptor were trivial. However, some HMS clients modify both table properties and storage descriptor. If table properties had non-trivial changes (e.g., a location change), skipping the file metadata reload led to inconsistencies.

The issue was addressed by refining the checks for skipping file metadata reload in ALTER_TABLE events.

Apache Jira: IMPALA-13403

CDPD-74861: Hive Iceberg commit error while running concurrent writes

In an Amazon AWS Environment, when Hive runs concurrent writes on the same Iceberg table, the job fails intermittently with an "Error committing job" message. The failure occurs in the MoveTask phase with error code 40000.

This issue is addressed by improving resource handling to prevent errors during query retries with a different YARN application ID.

Apache JIRA: HIVE-28649

CDPD-75422: Impala schema case sensitivity issue of Iceberg schema elements: Impala's schema is case insensitive, causing errors with mixed case schema elements created through Spark during predicate pushdown.; Case sensitivity issues with Impala schemas is resolved, ensuring compatibility with mixed case schema elements created through Spark.
Apache Jira: IMPALA-13463

CDPD-77713: Deadlock occurs in TxnStoreMutex when acquiring lock: Deadlocks occurred in Hive Metastore due to MySQL’s REPEATABLE-READ isolation level, which caused locking conflicts during housekeeping tasks.; The issue was addressed by restoring the TxnHandler's isolation level to READ-COMMITTED.
Apache Jira: HIVE-28669
CDPD-77905: MRCompactor causes data loss during major compaction: During a major compaction, records matching certain conditions were lost due to incorrect handling in MRCompactor.; The issue was addressed by ensuring that all records are correctly preserved during major compaction.
Apache Jira: HIVE-28700
ENGESC-28372: Unable to resize Workload Aware Autoscaling enabled Impala Virtual Warehouse using the UI: If you are using the Cloudera Data Warehouse UI to resize an Impala Virtual Warehouse that is enabled for Workload Aware Autoscaling, you may notice a message in the Sizing and Scaling tab of the Virtual Warehouse Details page — "Some operations are still running. Please wait...", and you are unable to proceed further although the Virtual Warehouse is in a healthy state.; This issue is now addressed and you can successfully resize a Workload Aware Autoscaling enabled Impala Virtual Warehouse.
IMPALA-12607: Optimized event fetching by filtering at the metastore: Impala fetched all metastore events and filtered them in its cache based on DbName/TableName. This approach became a bottleneck when handling a large number of events.; Impala now directly fetches events specific to the database or table from the metastore by leveraging the HIVE-27499 change.
Apache Jira: IMPALA-12607