LIVE_PROGRESS Query Option (CDH 5.5 or higher only)

For queries submitted through the impala-shell command, displays an interactive progress bar showing roughly what percentage of processing has been completed. When the query finishes, the progress bar is erased from the impala-shell console output.

Type: Boolean; recognized values are 1 and 0, or true and false; any other value interpreted as false

Default: false (shown as 0 in output of SET statement)

Command-line equivalent:

You can enable this query option within impala-shell by starting the shell with the --live_progress command-line option. You can still turn this setting off and on again within the shell through the SET command.

Usage notes:

The output from this query option is printed to standard error. The output is only displayed in interactive mode, that is, not when the -q or -f options are used.

For a more detailed way of tracking the progress of an interactive query through all phases of processing, see LIVE_SUMMARY Query Option (CDH 5.5 or higher only).

Restrictions:

Because the percentage complete figure is calculated using the number of issued and completed "scan ranges", which occur while reading the table data, the progress bar might reach 100% before the query is entirely finished. For example, the query might do work to perform aggregations after all the table data has been read. If many of your queries fall into this category, consider using the LIVE_SUMMARY option instead for more granular progress reporting.

The LIVE_PROGRESS and LIVE_SUMMARY query options currently do not produce any output during COMPUTE STATS operations.

Because the LIVE_PROGRESS and LIVE_SUMMARY query options are available only within the impala-shell interpreter:

  • You cannot change these query options through the SQL SET statement using the JDBC or ODBC interfaces. The SET command in impala-shell recognizes these names as shell-only options.

  • Be careful when using impala-shell on a pre-CDH 5.5 system to connect to Impala running on a CDH 5.5 or higher system. The older impala-shell does not recognize these query option names. Upgrade impala-shell on the systems where you intend to use these query options.

  • Likewise, the impala-shell command relies on some information only available in CDH 5.5 / Impala 2.3 and higher to prepare live progress reports and query summaries. The LIVE_PROGRESS and LIVE_SUMMARY query options have no effect when impala-shell connects to a cluster running an older version of Impala.

Added in: CDH 5.5.0 / Impala 2.3.0

Examples:

[localhost:21000] > set live_progress=true;
LIVE_PROGRESS set to true
[localhost:21000] > select count(*) from customer;
+----------+
| count(*) |
+----------+
| 150000   |
+----------+
[localhost:21000] > select count(*) from customer t1 cross join customer t2;
[###################################                                   ] 50%
[######################################################################] 100%

To see how the LIVE_PROGRESS and LIVE_SUMMARY query options work in real time, see this animated demo.