How to Configure Resource Management for Impala

Use Static Service Pools to allocate dedicated resources for Impala and other services to allow for predictable resource availability.

Static service pools isolate services from one another, so that high load on one service has bounded impact on other services. You can use Cloudera Manager to configure static service pools that control memory, CPU and Disk I/O.

The following screenshot shows a sample configuration for Static Service Pools in Cloudera Manager:

Within the constraints of the static service pool, you can further subdivide Impala's resources using Admission Control. You configure Impala Admission Control pools in the Cloudera Manager Dynamic Resource Pools page.

You use Admission Control to divide usage between Dynamic Resource Pools in multitenant use cases. Allocating resources judiciously allows your most important queries to run faster and more reliably.

Admission Control is enabled by default.

Use per-query memory limits to prevent queries from consuming excessive memory resources that impact other queries. Cloudera recommends that you set the query memory limits whenever possible.

If you set the Pool Max Mem Resources for a resource pool, Impala attempts to throttle queries if there is not enough memory to run them within the specified resources.

Only use admission control with maximum memory resources if you can ensure there are query memory limits. Set the pool Default Query Memory Limit to be certain. You can override this setting with the query option, if necessary.

Typically, you set query memory limits using the set MEM_LIMIT=Xg; query option. When you find the right value for your business case, memory-based admission control works well. The potential downside is that queries that attempt to use more memory might perform poorly or even be cancelled.

A dynamic resource pool is a named configuration of resources and a policy for scheduling the resources among Impala queries running in the pool. Dynamic resource pools allow you to schedule and allocate resources to Impala queries based on a user's access to specific pools and the resources available to those pools.

This example creates both production and development resource pools or queues. It assumes you have 3 worker nodes with 24GiB of RAM each for an aggregate memory of 72000MiB. This pool configuration allocates the Production queue twice the memory resources of the Development queue, and a higher number of concurrent queries.

To create a Production dynamic resource pool for Impala:

1. In Cloudera Manager, select Clusters > Dynamic Resource Pool Configuration.
2. Click the Impala Admission Control tab.
3. Click Create Resource Pool.
4. Specify a name and resource limits for the Production pool:
   • In the Resource Pool Name field, enter Production.
   • In the Max Memory field, enter 48000.
   • In the Default Query Memory Limit field, enter 1600.
   • In the Max Running Queries field, enter 10.
   • In the Max Queued Queries field, enter 200.
5. Click Create.
6. Click Refresh Dynamic Resource Pools.

The Production queue runs up to 10 queries at once. If the total memory requested by these queries exceeds 48000 MiB, it holds the next query in the queue until the memory is released. It also prevents a query from running if it needs more memory than is currently available. Admission Control holds the next query if either Max Running Queries is reached, or the pool Max Memory limit is reached.

Here, Max Memory resources and Default Query Memory Limit throttle throughput to 10 queries, so setting Max Running Queries might not be necessary, though it does not hurt to do so. Most users set Max Running Queries when they cannot pick good numbers for memory. Since users can override the query option mem_limit, setting the Max Running Queries property might make sense.

To create a Development dynamic resource pool for Impala:

1. In Cloudera Manager, select Clusters > Dynamic Resource Pool Configuration.
2. Click the Impala Admission Control tab.
3. Click Create Resource Pool.
4. Specify a name and resource limits for the Development pool:
   • In the Resource Pool Name field, enter Development.
   • In the Max Memory field, enter 24000.
   • In the Default Query Memory Limit field, enter 8000.
   • In the Max Running Queries field, enter 1.
   • In the Max Queued Queries field, enter 100.
5. Click Create.
6. Click Refresh Dynamic Resource Pools.

   The Development queue runs one query at a time. If the total memory required by the query exceeds 24000 GiB, the query is rejected and not executed.

Anne Chang is administrator for an enterprise data hub that runs a number of workloads, including Impala.

Anne has a 20-node cluster that uses Cloudera Manager static partitioning. Because of the heavy Impala workload, Anne needs to make sure Impala gets enough resources. While the best configuration values might not be known in advance, she decides to start by allocating 50% of resources to Impala. Each node has 128 GiB dedicated to each impalad. Impala has 2560 GiB in aggregate that can be shared across the resource pools she creates.

Next, Anne studies the workload in more detail. After some research, she might choose to revisit these initial values for static partitioning.

To set up admission control for this workload, Anne first runs the workloads independently, so that she can observe the workload’s resource usage in Cloudera Manager. If they could not easily be run manually, but had been run in the past, Anne uses the history information from Cloudera Manager. It can be helpful to use other search criteria (for example, user) to isolate queries by workload. Anne uses the Cloudera Manager chart for Per-Node Peak Memory usage to identify the maximum memory requirements for the queries.

Based on these observations, Anne creates the admission control configuration with the following pools: