Periodic Stacks Collection

Periodic stacks collection allows you to enable and configure the periodic collection of thread stack traces in Cloudera Manager. When stacks collection is enabled for a role, call stacks are output to a log file at regular intervals. The logs can help with diagnosis of performance issues such as deadlock, slow processing, or excessive numbers of threads.

Stacks collection may impact performance for the processes being collected as well as other processes on the host, and is turned off by default. For troubleshooting performance issues, you may be asked by Cloudera Support to enable stacks collection and send the resulting logs to Cloudera for analysis.

Stacks collection is available for the majority of roles in Cloudera Manager. For the HDFS service, for example, you can enable stacks collection for the DataNode, NameNode, Failover Controller, HttpFS, JournalNode, and NFS Gateway. If the Stacks Collection category does not appear in the role's configuration settings, the feature is not available for that role.

Configuring Periodic Stacks Collection

To enable and configure periodic stacks collection, open the Cloudera Manager page for a specific service or role. Access the configuration settings in one of the following ways:
  • From the service page in Cloudera Manager:
    • Click the Configuration tab.
    • Select Scope > NameNode.
    • Select Category > Stacks Collection.
  • From the service page in Cloudera Manager:
    • Click the Instances tab.
    • Click the Configuration tab.
    • Select Scope > role type.
    • Select Category > Stacks Collection.
The configuration settings are as follows:
  • Stacks Collection Enabled - Whether or not periodic stacks collection is enabled.
  • Stacks Collection Directory - The directory in which stack logs will be placed. If not set, stacks will be logged into a stacks subdirectory of the role's log directory.
  • Stacks Collection Frequency - The frequency with which stacks will be collected.
  • Stacks Collection Data Retention - The amount of stacks data that will be retained. When the retention limit is reached, the oldest data will be deleted.
  • Stacks Collection Method - The method that will be used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles with an HTTP server endpoint that exposes the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.
As an example, to configure stacks collection for an HDFS NameNode, perform the following steps:
  1. Go to the HDFS service page.
  2. Click the Configuration tab.
  3. Select Scope > NameNode.
  4. Select Category > Stacks Collection.
  5. Locate the property or search for it by typing its name in the Search box.
  6. Modify the configuration settings if desired.
  7. Click Save Changes.

Stacks collection configuration settings are stored in a per-role configuration file called cloudera-stacks-monitor.properties. Cloudera Manager reads the configuration file and coordinates stack collection. Changes to the configuration settings take effect after a short delay. It is not necessary to restart the role.

Viewing and Downloading Stacks Logs

Stacks are collected and logged to a compressed, rotated log file. A certain amount of the log data is in an uncompressed file. When that file reaches a limit, the file is rotated and bzip2 compressed. Once the total number of files exceeds the configured retention limit, the oldest files are deleted.

Collected stacks data is available for download through the Cloudera Manager UI and API. To view or download stacks logs through the UI, perform the following steps:
  1. On the service page, click the Instances tab.
  2. Click the role in the Role Type column.
  3. In the Summary section of the role page, click Stacks Logs.
  4. Click Stacks Log File to view the most recent stacks file. Click Download Stacks Logs to download a zipped bundle of the stacks logs.