Maintaining Navigator Audit Server

Cloudera Manager Required Role: Navigator Administrator (or Full Administrator)

When you have Cloudera Navigator running, Navigator Audit Server is enabled by default. Audits are enabled for all supported services, including HDFS, Hive, Impala, Hue, HBase, Sentry, and Solr as well as Cloudera Manager and Navigator itself. Events are retained for 90 days with some default filters defined.

Because the default settings may not work for all environments, we recommend that you review your Navigator Audit Server setup to review these three areas:

What events are collected?: Default filters remove some service-to-service events. However, you should check to make sure that the role names used in the filters match what's used in your system.
How long are events retained?: To determine the best expiration period for events in your audit system, consider the volume of incoming events, the disk space available for the audit database, and how your organization is using the audits from the Navigator console or API.
How are events archived?: Mot organizations keep audit events in storage well after they are useful for casual queries through the Navigator console. Your archiving solution should balance the effort required to access archived audit data with the cost of implementing your archiving system. Be sure that whatever system you design can handle the volume of audits you intend to maintain.

This Navigator YouTube video gives an overview of the audit tuning process and some database queries and other techniques that will help you make sure your audit system is working well.

Navigator Audit Checkup

Reviewing Navigator Audits for Unproductive Events

You can use the Navigator console, Audit tab to get a general feel for what events are audited and if you are seeing too many audit events that don't add value. In addition, here are some queries you can run against the audit database itself to identify users or operations that produce more events than expected or needed.

Review Audited Operations

Log into the database as a privileged user.
For example for MySQL:
```
$ mysql -u user_name -p
```
You are prompted for a password.
Show the databases and look for the Navigator database.
```
mysql> show databases;
mysql> use navigator audit server database
```
If you aren't sure of the name, you can check the Navigator Audit Server Database Name in Cloudera Manager.
List the tables to see that the table names show the service and the day the audits are from.
```
mysql> show tables;
```
Pick a table and run a query to show the events recorded by user.
```
mysql> select username, operation, count(*) as usercount from audit table group by username order by usercount DESC;
```
If you find that a relatively large number of events were collected for a single user, it may be that this user is a system user. The default filters discard events performed by the HDFS superuser, but only if the role name is hdfs. You may need to add another audit filter to discard events by this additional system user name.
Pick a table and run a query to show the events recorded by operation.
```
mysql> select operation, username, count(*) as opcount
              from audit table group by operation order by
              opcount DESC;
```
This query may indicate that your audit system includes large numbers of events that you may not care to track. In that case, consider adding an audit filter to discard the unneeded events by operation.

Reviewing Default Audit Filters

To see the default Navigator Audit Filters:

Log in to the Cloudera Manager Admin Console.
Select the service you want to review. Select Cluster > Service-Name
Click the Configuration tab.
Select Service (Service-Wide) for the Scope filter.
Select Cloudera Navigator for the Category filter.
Click for Audit Event Tracker to open the property description.

The make sure the default filters are doing what they are designed to do, check that the user names in the filters correspond to the role names in your environment. For example, one default filter is configured to discard redundant audit events produced in HDFS when operations are performed in Hive or other services. This filter is configured for the user hdfs. If your environment uses other names for HDFS superusers, consider creating additional filters to discard the same operations performed under the other superuser names.

Review and Predict Audit Volume

One way to calculate the volume of disk space you’ll need for the Navigator Audit Server database is to look at the event count for a single day or average of days and multiply that volume by the number of days in the audit event expiration period. You can modify the expiration period in Cloudera Manager, in the Navigator Audit Server Data Expiration Period property. The default value is 90 days.

Calculate Audit Volume

To see the size of all audit tables for a given date, query against a set of tables. For example, in MySQL:

mysql> SELECT table_name AS `Table`, round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB`  FROM information_schema.TABLES  WHERE table_schema = "navigator audit database" and table_name like "%YYYY_MM_DD";

Insert your Navigator Audit Database name and a valid audit date (YYYY_MM_DD format) in the query.

To see the total size of audits for each day for a give year of audits, query against the event tables in the database for a given year:

SELECT sum(round(((data_length + index_length) / 1024 / 1024), 2)) `Size in MB across all audits`, right(table_name,10) as dateset  FROM information_schema.TABLES  WHERE table_schema = "navigator audit database" and table_name like "%YYYY_%"  group by dateset;

Insert your Navigator Audit Database name and a valid audit year (YYYY format) in the query.

Calculate the expected size of your audit database by selecting a representative size for a day of audit data and multiplying that value by the number of days in the expiration period.
For example, if your system is generating about 5 GB of events per day and you are keeping the default 90 days of audits, you’ll need about 450 GB of space for the audit database.

Publishing Audit Events

Navigator Metadata Server Management