PARQUET_DICTIONARY_FILTERING Query Option (CDH 5.11 or higher only)
The PARQUET_DICTIONARY_FILTERING query option controls whether Impala uses dictionary filtering for Parquet files.
To efficiently process a highly selective scan query, when this option is enabled, Impala checks the values in the Parquet dictionary page and determines if the whole row group can be thrown out.
A column chunk is purely dictionary encoded and can be used by dictionary filtering if any of the following conditions are met:
- If the encoding_stats is in the Parquet file, dictionary filtering uses it to determine if there are only dictionary encoded pages (i.e. there are no data pages with an encoding other than PLAIN_DICTIONARY).
- If the encoding stats are not present, dictionary filtering looks at the encodings. The column is purely dictionary encoded if both of the conditions satisfy:
- PLAIN_DICTIONARY is present.
- Only PLAIN_DICTIONARY, RLE, or BIT_PACKED encodings are listed.
- Dictionary filtering works for the Parquet dictionaries with less than 40000 values if the file was written by or lower.
In the query runtime profile output for each Impalad instance, the NumDictFilteredRowGroups field in the SCAN node section shows the number of row groups that were skipped based on dictionary filtering.
Note that row groups can be filtered out by Parquet statistics, and in such cases, dictionary filtering will not be considered.
The supported values for the query option are:
- true (1): Use dictionary filtering.
- false (0): Do not use dictionary filtering
- Any other values are treated as false.
Type: Boolean
Default: true (1)
Added in: CDH 5.12.0 / Impala 2.9.0