PARQUET_READ_STATISTICS Query Option (CDH 5.11 or higher only)
The PARQUET_READ_STATISTICS query option controls whether to read statistics from Parquet files and use them during query processing.
Parquet stores min/max stats which can be used to skip reading row groups if they don't qualify a certain predicate. When this query option is set to true, Impala reads the Parquet statistics and skips reading row groups that do not match the conditions in the WHERE clause.
Impala supports filtering based on Parquet statistics:
- Of the numerical types for the old version of the statistics: Boolean, Integer, Float
- Of the types for the new version of the statistics (starting in IMPALA 2.8): Boolean, Integer, Float, Decimal, String, Timestamp
- For simple predicates of the forms: <slot> <op> <constant> or <constant> <op> <slot>, where <op> is LT, LE, GE, GT, and EQ
The PARQUET_READ_STATISTICS option provides a workaround when dealing with files that have corrupt Parquet statistics and unknown errors.
In the query runtime profile output for each Impalad instance, the NumStatsFilteredRowGroups field in the SCAN node section shows the number of row groups that were skipped based on Parquet statistics.
- true (1): Read statistics from Parquet files and use them in query processing.
- false (0): Do not use Parquet read statistics.
- Any other values are treated as false.
Type: Boolean
Default: true
Added in: CDH 5.12.0 / Impala 2.9.0