Using Substitution Variables for Flume Kerberos Principal and Keytab

Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

Flume instances running on clusters that use Kerberos for authentication require the Kerberos principal and keytab for the HDFS sink or HBase sink to be configured in the flume.conf file. Cloudera Manager generates Flume keytab files automatically. However, the location of the files is not known in advance, so Cloudera Manager provides two substitution variables to use for these Kerberos artifacts as shown in the table:
Kerberos Description Substitution variable
kerberosPrincipal Fully qualified principal name, such as service/user@DOMAIN $KERBEROS_PRINCIPAL
kerberosKeytab Location on the host to keytabs for kerberos principal $KERBEROS_KEYTAB
Use the Flume substitution variables for the principal name and the keytab file path on each host. Here are some usage examples:

HDFS Sink Example

The following example shows an HDFS sink configuration in the flume.conf file (the majority of the HDFS sink configuration options have been omitted):

agent.sinks.sink-1.type = HDFS
agent.sinks.sink-1.hdfs.kerberosPrincipal = flume/_HOST@YOUR-REALM.COM
agent.sinks.sink-1.hdfs.kerberosKeytab = /etc/flume-ng/conf/flume.keytab
agent.sinks.sink-1.hdfs.proxyUser = weblogs

The text below shows the same configuration options with the substitution variables:

agent.sinks.sink-1.type = hdfs
agent.sinks.sink-1.hdfs.kerberosPrincipal = $KERBEROS_PRINCIPAL
agent.sinks.sink-1.hdfs.kerberosKeytab = $KERBEROS_KEYTAB
agent.sinks.sink-1.hdfs.proxyUser = weblogs

HBase Sink Example

The following example shows an HBase sink configuration in the flume.conf file (the majority of the HBase sink configuration options have been omitted):

agent.sinks.sink-1.type = hbase
agent.sinks.sink-1.kerberosPrincipal = flume/_HOST@YOUR-REALM.COM
agent.sinks.sink-1.kerberosKeytab = /etc/flume-ng/conf/flume.keytab

The text below shows the same configuration options with the substitution variables:

agent.sinks.sink-1.type = hbase
agent.sinks.sink-1.kerberosPrincipal = $KERBEROS_PRINCIPAL
agent.sinks.sink-1.kerberosKeytab = $KERBEROS_KEYTAB

Configuring Flume Substitution Variables Using Cloudera Manager Admin Console

Complete the following steps to have Cloudera Manager add these variables to the flume.conf file on every host that Cloudera Manager manages.

  1. Log in to the Cloudera Manager Admin Console.
  2. Select Clusters > Flume service.
  3. Click the Configuration tab.
  4. Select Agent for the Scope filter.
  5. In the Configuration File property, set the agent sinks used by your cluster equal to the substitution variables. For example, for an HDFS sink, the values would be specified as follows: 
    agent.sinks.sink-1.type = hdfs
agent.sinks.sink-1.hdfs.kerberosPrincipal = $KERBEROS_PRINCIPAL
agent.sinks.sink-1.hdfs.kerberosKeytab = $KERBEROS_KEYTAB
agent.sinks.sink-1.hdfs.proxyUser = weblogs
  6. Click Save Changes.