Known issues for the Cloudera Data Services on premises 1.5.5 SP1

You must be aware of the known issues and limitations, the areas of impact, and workaround in Cloudera Data Services on premises 1.5.5 SP1 release.

Cloudera Data Services on premises 1.5.5 existing known issues are carried into Cloudera Data Services on premises1.5.5 SP1. For more details, see Known Issues.

Known issue identified in 1.5.5 SP1

The following are the known issues identified in 1.5.5 SP1:

COMPX-20437 - DB connection failures causing RPM and CAM pods to CrashLoopBackOff: During an upgrade from version 1.5.5 to any 1.5.5 hotfix release, the cluster-access-manager (CAM) and resource-pool-manager (RPM) pods can enter a CrashLoopBackOff state if they are not automatically restarted during the upgrade.; After upgrade, manually restart the CAM pod and then restart the RPM pod (order of restart is very important).
Commands to restart:
kubectl rollout restart deployment <cluster-access-manager-deployment-name> -n <namespace> kubectl rollout restart deployment <resource-pool-manager-deployment-name> -n <namespace>

OPSX-4656: DRS automatic backup policy is inconsistent post upgrade: When you update a backup policy manually, the modifications to the policy are not persisted after a Control Plane upgrade.; Ensure that you manually update the policy after the upgrade activity is complete.

OBS-9491 - Prometheus configuration exceeds size limit in large environments: In environments with a large number of namespaces (approximately 300 or more per environment), the Prometheus configuration for Cloudera Monitoring might exceed the 1 MB Kubernetes Secret size limit. If the total size, which depends on factors such as the number of namespaces, the length of namespace names, their variability, and the size of the certificate store, exceeds 1 MB, the new Prometheus configuration will not be applied, and new namespaces will not be monitored. As a result, the telemetry data will not be collected from those namespaces and will not be reflected on the corresponding Grafana charts.; To resolve this issue, you must enable Prometheus configuration compression at the control plane level.

Upgrade to Cloudera Data Services on premises 1.5.5 SP1 or a higher version.

Set the environment variable ENABLE_ENVIRONMENT_PROMETHEUS_CONFIG_COMPRESSION to "true" on the cdp-release-monitoring-pvcservice deployment in the Cloudera Control Plane namespace.

note
The upper bound for monitoring is based on the number of namespaces per environment, the length of the namespace name, the variability of the names, and the size of the cert store.

For example, about 700 workload namespaces in a single environment with a short, consistent pattern ("aaaa-bbbb-XXXX") generate around 150 KB of configuration, which fits within the 1 MB limit.

About 3000 namespaces in a single environment with a longer, consistent pattern ("aaaa-cccc-dddd-XXXXX") generate about 900 KB.

OPSX-6618 - In Cloudera Embedded Container Service upgrade not all volumes are upgraded to the latest longhorn version: During restart of the Cloudera Embedded Container Service cluster from 1.5.5 to 1.5.5 SP1, the upgrade failed due to longhorn health issues. This is because one of the volumes was degraded.; Follow the steps to resolve the issue:

Identify the problematic volumes (degraded state).

Set the value of spec.numberOfReplicas of the volume to the number of active replicas. For example, set the value to 3 if two replicas are active.

Apply the fix before or during the upgrade as per the workaround instructions (refer to Longhorn issue #11825). Longhorn Engineering is addressing in v1.11.0 and will back-port the fix. The workaround is included as part of the upgrade. However, if the issue is noticed even after upgrade please follow the workaround steps documented here: https://github.com/longhorn/longhorn/issues/11825

OPSX-6566 - Cloudera Embedded Container Service restart fails with etcd connectivity issues: Restart of the Cloudera Embedded Container Service server fails with etcd error: "error reading from server"; Identify the server role that failed.

Restart only the Cloudera Embedded Container Service server role which failed.

Once the server role is restarted and healthy, proceed with the remainder of the Cloudera Embedded Container Service server role restart sequence.

OPSX-6401 - Istio ingress-default-cert is not created in the upgrade scenario: After upgrading to 1.5.5 SP1, the Secret ingress-default-cert is not created in the istio-ingress namespace. Because this certificate is expected, failing to create it causes components like CAII & MR provisioning to fail.; To resolve this issue:

After upgrading from an older version to 1.5.5 SP1, check for the secret called ingress-default-cert in the istio-ingress namespace (ns).
For example:
kubectl get secret ingress-default-cert -n istio-ingress

If missing, copy the identical secret from kube-system namespace (or from the pre-upgrade environment) into istio-ingress namespace with the same name and contents.

This will restore the expected certificate and allow CAII & MR provisioning to proceed.

OPSX-6645 - Cloudera Embedded Container Service upgrade failure at restart step: When the Cloudera Embedded Container Service role fails to start after a node reboot or service restart, the root cause can be that the etcd defragmentation process which runs on startup takes longer than the component timeout thresholds. As a result:

The kubelet service may fail to start or time out.

The kube-apiserver, kube-scheduler or kube-controller-manager roles may remain in a NotReady state.

etcd may perform automatic defragmentation at startup.

The API server may fail to connect to etcd (connection refused or timeout).; To fix this issue:

Resume the Cloudera Embedded Container Service start/restart from the Cloudera Manager UI once etcd has come up.

Ensure etcd meets production hardware and configuration requirements: For more information, see etcd requirements and Knowledge Base.

OPSX-6767 - Cloudera Embedded Container Service cluster has stale configuration after Cloudera Manager upgrade to 7.13.1.501-b2 from 7.11.3.24: After upgrading Cloudera Manager to version 7.13.1.501, the Cloudera Embedded Container Service shows a staleness indicator. This occurs due to configuration changes applied by the upgrade:

worker-shutdown-timeout: reduced from 24 hours (86,400 s) to 15 minutes (900 s).

smon_host: a new monitoring configuration added.

smon_port: a new monitoring port configuration (9997).; No action required — the staleness is expected following this upgrade and can be safely ignored. The indicator will automatically clear once Cloudera Embedded Container Service is upgraded to version 1.5.5 SP1 or later.
If you prefer to clear the staleness indicator right away, you may manually refresh the Cloudera Embedded Container Service service through the Cloudera Manager UI.
note
Performing a refresh will restart the ingress controller and may cause a brief service interruption.

OPSX-6638 - Post rolling restart many pods are stuck in pending state: Pods remain in Pending state and fail to schedule with etcd performance warnings.; Restart the Cloudera Embedded Container Service master role which has etcd performance issues. For more information, see etcd hardware recommendations and Knowledge Base.