Enabling Audit and Log Collection for Services

Cloudera Manager Required Role: Navigator Administrator (or Full Administrator)

Auditing of every service and role in the cluster may not be necessary and may degrade performance, which is why auditing is not always enabled by default. In addition, auditing can be configured to capture only specific events, as detailed in Configuring Service Auditing Properties.

Continue reading:

Enabling Audit Collection

Any service or role instance that can be audited by Cloudera Navigator has an Enable Audit Collection property. When enabled, the Cloudera Manager Agent process on the node monitors the audit log file (or files) and sends collected audit records to the Navigator Audit Server.

  1. Log in to Cloudera Manager Admin Console
  2. Select Clusters > Cloudera Management Service.
  3. Click the Configuration tab.
  4. Select ServiceName (Service-Wide) for the Scope filter.
  5. Select Navigator Metadata Server for the Category filter.
  6. Click the Enable Audit Collection checkbox to activate auditing for the service.
  7. Click Save Changes.
  8. Restart the service.

Configuring Impala Daemon Logging

To enable logging for the Impala Daemon role.
  1. Log in to Cloudera Manager Admin Console
  2. Select Clusters > Cloudera Management Service.
  3. Click the Configuration tab.
  4. Select Impala Daemon for the Scope filter.
  5. Select Logs for the Category filter.
  6. Edit the Enable Impala Audit Event Generation.
  7. Click Save Changes.
  8. Restart the Impala daemon.
To set the log file size:
  1. Click the Impala service.
  2. Select Scope > Impala Daemon.
  3. Select Category > Logs.
  4. Set the Impala Daemon Maximum Audit Log File Size property.
  5. Click Save Changes.
  6. Restart the Impala service.

Enabling Solr Auditing

Solr auditing is disabled by default. To enable auditing:
  1. Enable Sentry authorization for Solr following the procedure in Enabling Sentry Policy File Authorization for Solr.
  2. Go to the Solr service.
  3. Click the Configuration tab.
  4. Select Scope > Solr Service (Service-Wide)
  5. Select Category > Policy File Based Sentry category.
  6. Select or clear the Enable Sentry Authorization checkbox.
  7. Select Category > Cloudera Navigator category.
  8. Select or clear the Enable Audit Collection checkbox. See Configuring Service Audit Collection and Log Properties.
  9. Click Save Changes to commit the changes.
  10. Restart the service.

Configuring Audit Logs

The following properties apply to an audit log file:
  • Audit Log Directory - The directory in which audit log files are written. By default, this property is not set if Cloudera Navigator is not installed.

    A validation check is performed for all lifecycle actions (stop/start/restart). If the Enable Collection flag is selected and the Audit Log Directory property is not set, the validator displays a message that says that the Audit Log Directory property must be set to enable auditing.

    If the value of this property is changed, and service is restarted, then the Cloudera Manager Agent will start monitoring the new log directory for audit events. In this case it is possible that not all events are published from the old audit log directory. To avoid loss of audit events, when this property is changed, perform the following steps:

    1. Stop the service.
    2. Copy audit log files and (for Impala only) the impalad_audit_wal file from the old audit log directory to the new audit log directory. This needs to be done on all the hosts where Impala Daemons are running.
    3. Start the service.
  • Maximum Audit Log File Size - The maximum size of the audit log file before a new file is created. The unit of the file size is service dependent:
    • HDFS, HBase, Hive, Hue, Navigator Metadata Server, Sentry, Solr - MiB
    • Impala - lines (queries)
  • Number of Audit Logs to Retain - Maximum number of rolled over audit logs to retain. The logs will not be deleted if they contain audit events that have not yet been propagated to the Audit Server.
To configure audit logs do the following:
  1. Do one of the following:
    • Service - Go to a supported service.
    • Navigator Metadata Server
      • Do one of the following:
        • Select Clusters > Cloudera Management Service.
        • On the Home > Status tab, in Cloudera Management Service table, click the Cloudera Management Service link.
  2. Click the Configuration tab.
  3. Select the scope according to the service:
    • All services except Impala - Scope > ServiceName (Service-Wide).
    • Impala - Scope > Impala Daemon.
    • Navigator Metadata Server - Scope > Navigator Metadata Server.
  4. Select Category > Logs.
  5. Configure the log properties. For Impala, preface each log property with Impala Daemon.
  6. Click Save Changes.
  7. Restart the service.