Choosing and Configuring Data Compression

Guidelines for Choosing a Compression Type

• GZIP compression uses more CPU resources than Snappy or LZO, but provides a higher compression ratio. GZip is often a good choice for cold data, which is accessed infrequently. Snappy or LZO are a better choice for hot data, which is accessed frequently.
• BZip2 can also produce more compression than GZip for some types of files, at the cost of some speed when compressing and decompressing. HBase does not support BZip2 compression.
• Snappy often performs better than LZO. It is worth running tests to see if you detect a significant difference.
• For MapReduce and Spark, if you need your compressed data to be splittable, BZip2 and LZO formats can be split. Snappy and GZip blocks are not splittable, but files with Snappy blocks inside a container file format such as SequenceFile or Avro can be split. Snappy is intended to be used with a container format, like SequenceFiles or Avro data files, rather than being used directly on plain text, for example, since the latter is not splittable and cannot be processed in parallel. Splittability is not relevant to HBase data.
• For MapReduce, you can compress either the intermediate data, the output, or both. Adjust the parameters you provide for the MapReduce job accordingly. The following examples compress both the intermediate data and the output. MR2 is shown first, followed by MR1.
  • MRv2

    hadoop jar hadoop-examples-.jar sort "-Dmapreduce.compress.map.output=true"
    "-Dmapreduce.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec"
    "-Dmapreduce.output.compress=true"
    "-Dmapreduce.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec" -outKey
    org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text input output

  • MRv1

    hadoop jar hadoop-examples-.jar sort "-Dmapred.compress.map.output=true"
    "-Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec"
    "-Dmapred.output.compress=true"
    "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec" -outKey
    org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text input output