Fixed issues in 7.1.9 SP1 CHF 10

Know more about the cumulative hotfix 10 for 7.1.9 SP1.

Following are the list of fixes that were shipped for CDP Private Cloud Base version 7.1.9-1.cdh7.1.9.p1054.69605909.

CDPD-85295: Migrate commons-configuration to 2.x

The commons-configuration library is now upgraded in Zeppelin to address security issues and bugfixes.

CDPD-84529: Upgrade Commons BeanUtils to 1.11.0

The Commons BeanUtils library is now upgraded in Zeppelin to address security issues and bugfixes.

CDPD-82056: UI: when server response date fields as '0', UI shows as current time

Previously, if an API response contained an invalid date value, such as 0, intended for display on the user interface, the current system date was shown. Now, instead of displaying the system date when an invalid date value is encountered in the API response, the user interface displays NA. This issue specifically affects the Entity Detail page, where the create time and modified time are displayed.

Apache JIRA: ATLAS-5015

CDPD-88578: Resolved Missing Python Packages in Hue

Previously, Python packages were missing, leading to failures in Hue operations. This issue is now resolved, as the Hue environment now includes all necessary dependencies, eliminating the previously encountered errors.

CDPD-72557: Use Ozone Manager's KMS from client side when connecting to a cluster and dealing with encrypted data

Ozone client now uses server side Key Management Server (KMS) settings by default to encrypt or decrypt encryption keys with the cluster KMS server.

Apache JIRA: HDDS-11227

CDPD-84518: Expired secret key can abort leader OM startup

Previously, Ozone Manager (OM) start failure issues were caused by Null Pointer Exception (NPE) because of expired SCM managed secret keys. This issue is now resolved.

Apache JIRA: HDDS-13234

CDPD-66713: HDDS-9388 broke encryption

Previously, a file was not encrypted in an encrypted bucket caused by HDDS-9388 changes. This issue is now resolved.

Apache JIRA: HDDS-10363

CDPD-83223: Validate Container Balancer move timeout and replication timeout configurations

The move.timeout - move.replication.timeout - event.timeout.datanode.offset rule is now added for container balancer configurations. The timeout value must at least be 9 minutes.

Apache JIRA: HDDS-13068

CDPD-83224: Container Balancer delete commands are sent with an expiration time in the past

Previously, delete commands were getting discarded in the container balancer due to incorrect expiration time. With this fix, the container balancer can now send delete commands correctly.

Apache JIRA: HDDS-13067

CDPD-87933: DeleteBlocksCommandHandler thread stop for normal exception

Previously, block deletion was stopped due to invalid volume and container content, which resulted in continuous retry by the Storage Container Manager (SCM). This issue is now fixed.

Apache JIRA: HDDS-13448

CDPD-64642: DataNode Restart stuck in irrecoverable state with multiple exceptions

Previously, invalid container whose db path was missing in the memory container set, caused Null Pointer Exception for operations when accessing the container. This issue is now fixed.

Apache JIRA: HDDS-10061

CDPD-62073: OM Ratis Write - Move ACL check and Bucket resolution to preExecute

Previously, inconsistent behavior of Access Control List (ACL) validation on Ozone Manager (OM) nodes in high availability (HA) due to environmental issues could result in different results of data in those OMs and could have inconsistent data in OMs in HA. This issue is now fixed.

Apache JIRA: HDDS-9388

CDPD-77054: Exporting shell entities can cause the import to fail

Previously, when exporting Apache Hive tables created with Apache Spark, shell entities could be present in Apache Atlas. If these unresolved shell entities were included in the export zip file, the subsequent import operation failed.

This issue is now fixed. The import process now ensures that table entities are properly initialized and marked as complete, even if shell entities are present. This prevents failures during import when unresolved shell entities exist in the export.

CDPD-84506: Advanced Search is not working properly in the Atlas UI

Fixed an issue in the Atlas UI where Advanced Search did not trigger API calls for each relation. The UI now ensures that API calls are made for all relevant relations, providing accurate and complete search results.

This update also incorporates improvements from CDPD-65619, which enhanced the reliability and consistency of the Advanced Search operations in the Atlas UI.

COMPX-21359: Capacity Scheduler UI queue filter is not working as expected

The Capacity Scheduler UI queue filter did not work as expected when submitting apps with leaf queue's name. This issue is now resolved and the Replication Manager now returns the full queue path regardless of whether the application was submitted with a leaf queue name or a full queue path.

Apache JIRA: YARN-11662

CDPD-83781: Upgraded Jetty version to 9.4.57.v20241219

The Jetty version is upgraded to 9.4.57.v20241219 due to CVE-2024-8184 and CVE-2024-13009.

CDPD-83429: Logic to check if metric collection is thread-safe or not, not to parse the entire configuration every-time

Previously, the KMSWenApp metric collection logic tried getting and parsing the Key Management Server (KMS) configuration every time KMS configuration was required. The underlying configuration class acquired a lock for initialization blocking other threads from using the KMS configuration.

This issue is now fixed. The configuration value is now initialized and kept as an instance member, which is used every time to check metrics collection thread safety flags.

CDPD-88100: Reduced Impala log verbosity for query details

Impala logs were excessively verbose, particularly for Remote Procedure Calls (RPCs) that fetched query runtime profiles and execution summaries.

This issue is now resolved by lowering the logging level for these RPCs to VLOG_QUERY (or VLOG(1))

Apache Jira: IMPALA-13556

CDPD-85268: Stable table renames during metadata invalidations

A system error previously occurred when renaming a table while also invalidating its metadata.

The issue is now resolved by fixing NullPointerException in Catalogd that may occur when renaming a table while also invalidating its metadata.

Apache Jira: IMPALA-13631

CDPD-83645: RSASSA-PSS signature algorithm support in kRPC

kRPC communication previously failed when certificates used the RSASSA-PSS signature algorithm.

This issue is now resolved by updating the code (through KUDU-3663) to correctly determine the hash algorithm for RSASSA-PSS signatures.

Apache Jira: IMPALA-14038

CDPD-83215: Consistent user identity with cookie authentication

When using Impala with proxy clients, such as Knox, and cookie-based authentication, user requests sometimes failed.

This issue is now resolved by ensuring that the Kerberos user identity is correctly set in the connection context for sessions authenticated through cookies.

Apache Jira: IMPALA-14083

CDPD-87887: Results for HWC direct read V2 mode

After upgrading to CDP 7.1.7, reading tables with Hive Warehouse Connector (HWC) in DIRECT_READ_V2 mode returned incorrect data, as deletes were not applied and updates showed duplicated rows.

This issue is now resolved by reverting a problematic change and applying a new correction. This restores the correct behavior of the DIRECT_READ_V2 mode, ensuring accurate data is returned after update or delete operations.

CDPD-83679: Stable ARRAY(*) function for mixed column types

Users previously encountered a system error when using the ARRAY(*) function on tables combining string and boolean column types. This occurred because the system failed to correctly determine a common data type for these mixed columns.

This issue is now resolved by ensuring that the system correctly identifies string as the common data type when processing mixed string and boolean columns. This prevents the previous error and allows the ARRAY(*) function to work as expected.

Apache Jira:HIVE-28815

CDPD-65287: Stable Hive client close operations

Hive JDBC clients sometimes failed when closing operations, especially after network timeouts or retries. This happened because a duplicate close request arrived after the operation was already completed.

This issue is now resolved by allowing the JDBC client to ignore duplicate close requests that arrive after an operation is already completed.

Apache Jira:HIVE-25509

CDPD-84215: DayofWeek UDF results in consistent results irrespective of UTC / non-UTC timezones

The dayofweek() function previously returned inconsistent results for non-UTC timezones when the vectorized execution path was used.

This issue is now resolved by modifying the internal implementation of the dayofweek() user-defined function (UDF) for the Vectorized path execution in a way that now the date is not converted to UTC timezone before the UDF logic implementation.

Apache Jira:HIVE-28075

CDPD-84531: Connection starvation in HMS if datanucleus value generation fails

Database connections were sometimes not released as expected when internal processes failed.

This issue is now resolved to ensure continuous and reliable Hive Metastore operation by upgrading the datanucleus-rdbms component to version 5.2.13.

Apache Jira: HIVE-28839

CDPD-85086/CDPD-85497: Consistent property application for compaction jobs

When you set special rules for Hive compaction jobs, such as how to access logs, those rules weren't always applied to the part that updates statistics. This meant you could not see important logs for fixing problems, and your special settings were not fully used.

This issue is now resolved by making sure that all the special rules you set for compaction jobs are now also applied to the StatsUpdater job. As a result, you can easily check compaction logs for troubleshooting, and all your custom settings work as expected.

Apache Jira: Apache Jira: HIVE-28947

CDPD-83683: String conversion for complex data types

Applying functions such as md5() to complex data types previously caused errors because these types could not be directly converted to strings.

This issue is resolved by allowing complex data types to be converted to strings through casting. Now, functions such as md5() will work correctly with your complex data.

Apache Jira: HIVE-28825

CDPD-85723: IllegalStateException with Iceberg table with DELETE

Running a query on an Iceberg table fails with an IllegalStateException error in the following scenario:

• The Iceberg table has delete files for every data file (no data files without delete files) AND
• An anti-join operation is performed on the result of the Iceberg delete operation (IcebergDeleteNode or HashJoinNode)

This fix resolves the issue by setting the TableRefIds of the node corresponding to the Iceberg delete operation (`IcebergDeleteNode` or `HashJoinNode`) to only the table reference associated with the data files, excluding the delete files.

Apache Jira: IMPALA-14154

CDPD-88148: LEFT ANTI JOIN fails on Iceberg V2 tables with Delete files

Queries using a LEFT ANTI JOIN fail with an AnalysisException if the right-side table is an Iceberg V2 table containing delete files. For example, consider the following query:

SELECT * FROM table_a a
LEFT ANTI JOIN iceberg_v2_table b
ON a.id = b.id;

The error Illegal column/field reference'b.input_file_name' of semi-/anti-joined table 'b' is displayed because semi-joined tuples need to be explicitly made visible for paths pointing inside them to be resolvable.

The fix updates the IcebergScanPlanner to ensure that the tuple containing the virtual fields is made visible when it is semi-joined.

Apache Jira: IMPALA-13888

CDPD-87719: Ozone snapshot diff command is failing

The snapdiff operation could fail with a Null Pointer Exception if a directory had metadata changes such as Access Control List (ACL) updates or modification time updates between snapshots. This issue is now resolved.

CDPD-87776: Recon - Retrigger of build whole NSSummary tree task submission inconsistency

Executor may not run scheduled task or remove executor service:

org.apache.hadoop.ozone.recon.ReconUtils#triggerRebuild

Apache Jira: HDDS-13428

CDPD-59790: Fix LDB CLI for Ozone Snapshot DB

The Ozone Debug LDB CLI did not support any Ozone Snapshot databases. This issue is not resolved.

Apache Jira: HDDS-9119

CDPD-66744: Set Times API does not work with linked buckets

The setTimes API that operated on a linked bucket failed because of a bug. This issue is now resolved.

Apache Jira: HDDS-10369

CDPD-88172: Make ozone.snapshot.filtering.service.interval reconfigurable

ozone.snapshot.filtering.service.interval can be changed and reconfigured using the ozone admin reconfig CLI to enable or disable the SstFilteringService or change the internal for which the service is run.

Apache Jira: HDDS-13464

CDPD-88179: Exception handling for unchecked exception for deleteBlock command from SCM

This fix provides improved exception handling for unchecked exception for deleteBlock command from SCM.

Apache Jira: HDDS-13451

CDPD-87270: dt configuration dependency on secert key configuration

Adjusted the secret key expiry calculation (hdds.secret.key.expiry.duration = 9d) to account for delegation token lifetimes. This change ensures that tokens remain valid for their full configured duration, preventing premature key invalidation and improving stable authentication.

Apache Jira: HDDS-13343

CDPD-87378: RangerOzoneAuthorizer displays a Null Pointer Exception (NPE) when multiple instances of the classes are initialized

Previously, a NPE can be thrown during Ozone Manager re-initiailization while Ozone Manager automatic bootstrap takes place. The follower Ozone Manager can fail to apply a transaction in such a case. This issues is now fixed.

Apache Jira: HDDS-11472

CDPD-74753: Improvements for large scale deletion

In this release, to speed up ozone data deletion, multiple issues are handled. The fixes under this issue focus on several critical areas, including the following areas:

• Performance Improvements

  Improving speed that space is reclaimed from the system. Optimization of deletion processes to handle large-scale operations more efficiently.

• Bug Fixes

  Addressing issues that occur specifically when deleting large amounts of data.

• Observability Enhancements

  Improving observability throughout the deletion process through logs, metrics, and dashboards. New metrics have been introduced and added to Grafana, with a lightweight dashboard created to track deletion progress.

Apache JIRA: HDDS-11605, HDDS-11712, HDDS-11713, HDDS-11498, HDDS-8888, HDDS-8882, HDDS-12087, HDDS-11633, HDDS-4368, HDDS-11509, HDDS-11711, HDDS-11779, HDDS-12238, HDDS-11511, HDDS-11512, HDDS-12442, HDDS-12474, HDDS-11507, HDDS-12320, HDDS-12175, HDDS-10407, HDDS-10386, HDDS-12975, HDDS-10561, HDDS-13103, HDDS-13057, HDDS-13259, HDDS-13142, HDDS-13079, HDDS-13080, HDDS-8869, HDDS-10705, HDDS-13489

CDPD-84567: Backport upstream fixes for missing blocks issue

Backported the following upstream fixes:

Apache JIRA: HDFS-15764, HDFS-16985, HDFS-17342