SCHEDULE_RANDOM_REPLICA Query Option (CDH 5.7 or higher only)

The SCHEDULE_RANDOM_REPLICA query option fine-tunes the algorithm for deciding which host processes each HDFS data block. It only applies to tables and partitions that are not enabled for the HDFS caching feature.

Type: Boolean; recognized values are 1 and 0, or true and false; any other value interpreted as false

Default: false

Added in: CDH 5.7.0 / Impala 2.5.0

Usage notes:

In the presence of HDFS cached replicas, Impala randomizes which host processes each cached data block. To ensure that HDFS data blocks are cached on more than one host, use the WITH REPLICATION clause along with the CACHED IN clause in a CREATE TABLE or ALTER TABLE statement. Specify a replication value greater than or equal to the HDFS block replication factor.

The SCHEDULE_RANDOM_REPLICA query option applies to tables and partitions that do not use HDFS caching. By default, Impala estimates how much work each host has done for the query, and selects the host that has the lowest workload. This algorithm is intended to reduce CPU hotspots arising when the same host is selected to process multiple data blocks, but hotspots might still arise for some combinations of queries and data layout. When the SCHEDULE_RANDOM_REPLICA option is enabled, Impala further randomizes the scheduling algorithm for non-HDFS cached blocks, which can further reduce the chance of CPU hotspots.

This query option works in conjunction with the work scheduling improvements in CDH 5.7 / Impala 2.5 and higher. The scheduling improvements distribute the processing for cached HDFS data blocks to minimize hotspots: if a data block is cached on more than one host, Impala chooses which host to process each block based on which host has read the fewest bytes during the current query. Enable SCHEDULE_RANDOM_REPLICA setting if CPU hotspots still persist because of cases where hosts are "tied" in terms of the amount of work done; by default, Impala picks the first eligible host in this case.

Related information:

Using HDFS Caching with Impala (CDH 5.3 or higher only), Avoiding CPU Hotspots for HDFS Cached Data , REPLICA_PREFERENCE Query Option (CDH 5.9 or higher only)