Flume MorphlineInterceptor Configuration Options

Flume can modify and drop events using Interceptors, which can be attached to any Flume source. The MorphlineInterceptor runs morphline transformations on intercepted events. For example, a morphline can ignore events or alter or insert certain event headers using regular expression-based pattern matching, or it can auto-detect and set a MIME type using Apache Tika. Interceptors can be used for content-based routing in a Flume topology.

Flume also supports multiplexing event flows by defining a flow multiplexer that can replicate or selectively route an event to specific channels. The Flume User Guide example demonstrates a source from agent foo fanning out the flow to three different channels. This can be a replicating or multiplexingfan-out. In a replicating fan-out, each event is sent to all three channels. In a multiplexing fan-out, an event is sent to a subset of available channels when an event attribute matches a specified value. For example, if an event attribute called stream.type is set to application/pdf, it goes to channel1 and channel3. If the attribute is set to avro/binary, it goes to channel2. If a channel is unavailable, then an exception is thrown, and the event is replayed after the channel becomes available again. You can configure this mapping in the flume.conf file.

The MorphlineInterceptor supports the following configuration options (required options in bold):

Property Name Default Description
type   Must be set to the fully qualified class name (FQCN) org.apache.flume.sink.solr. morphline.MorphlineInterceptor$Builder.
morphlineFile   The location of the morphline configuration file.
  • For Cloudera Manager deployments, use: 
    <agentName>.sources.<sourceName>.interceptors.
<interceptorName>.morphlineFile = morphlines.conf
  • For unmanaged deployments, provide the relative or absolute path on the local filesystem to the morphline configuration file. For example, /etc/flume-ng/conf/morphline.conf.
morphlineId null The name of the morphline to use when there is more than one morphline in a morphline configuration file.
This example shows a section for a MorphlineInterceptor named morphlineinterceptorfor a source named avroSrc for an agent named agent: 
agent.sources.avroSrc.interceptors = morphlineinterceptor
agent.sources.avroSrc.interceptors.morphlineinterceptor.type = org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
agent.sources.avroSrc.interceptors.morphlineinterceptor.morphlineFile = /etc/flume-ng/conf/morphline.conf
agent.sources.avroSrc.interceptors.morphlineinterceptor.morphlineId = morphline1