Managing Disk Space for Log Files

All CDH cluster hosts write out separate log files for each role instance assigned to the host. Cluster administrators can monitor and manage the disk space used by these roles and configure log rotation to prevent log files from consuming too much disk space.

Disk Space Requirements

For each role assigned to a host, you should generally provision 2GB of disk space for log files. This recommendation is based on the default values of configuration properties that set the maximum log file size (200MB) and the maximum number of files (10). To calculate the disk space required for each host, multiply the configured maximum size of the log file by the configured maximum number of logs. Perform this calculation for each role on a host and add them together. (Note that Gateway roles do not generate log files.)

To determine the roles assigned to each host, open the Cloudera Manager Admin Console and go to Hosts > All Hosts and expand the list of roles in the Roles column.

Managing Log Files

To manage log file configurations for all role instances of a service:
  1. Go to Service Name > Configuration.
  2. Select Category > Logs.
  3. Edit the logging parameters.
  4. Click Save Changes.
There are the parameters you use to manage log files:
Log File Properties
Property Description Default Value
Role Type Max Log Size Maximum size for a log file before the log file rolls over into a new file. 200 MB
Role Type Maximum Log File Backups Maximum number of rolled-over log files to retain. 10
Role Type Log Directory

The path to the directory where the log files are saved.

 /var/log/log_file_name
Role Type Logging Threshold

(not available for all roles)

 Logging level to limit the number of entries saved in the log file. Depends on the role.