Fluentd Blog

Fluentd v0.12 is Released

We are pleased to announce that we have released Fluentd v0.12 today. v0.12 introduces nice features and improves internal structures (in anticipation for v1).

RubyGems: fluentd 0.12.0

New Features

Configuration

--use-v1-config is used by default, which means that the user can use JSON arrays/hashes and embedded Ruby code natively. See the docs for further details.

In addition, we introduced the @ prefix for system parameter or label symbols to avoid conflicts with plugin parameters.

• @type specifies plugin type
• @id specifies plugin id. in_monitor_agent uses this value for plugin_id field.
• @label specifies label. See the Label section below.

type and id are still valid to ensure backward compatibility.

A Small Syntactical Incompatibility

v1 config has one incompatibility problem: if the value of a parameter starts with the characters '{' or '[', it is now interpreted as JSON. To keep using the existing config, the parameter value must be explicitly quoted. Check "Multi line support for array and hash values" for further details.

Filter

v0.12 introduces filter plugins to process event streams. With filter plugins, the old trick of "using output plugins as filter" is no longer needed.

Seeing is believing:

<filter pattern1>
  @type filter_foo
  # ...
</filter>
<filter pattern2>
  @type filter_bar
  # ...
</filter>

Fluentd tries to apply a filter chain to event streams. If a tag is matched with pattern1 and pattern2, Fluentd applies filter_foo and filter_bar top-to-bottom (filter_foo followed by filter_bar). See this v0.12 configuration as a detailed example.

v0.12 ships with grep and record_transformer plugins.

• grep

The grep filter is a filter version of fluent-plugin-grep of output plugin. The configuration example is below:

<filter foo.**>
  @type grep
  regexp1 message INFO
</filter>

If the value of the "message" field doesn't match "INFO", such events are removed from event stream. grep filter doesn't support the old input_key and regexp pair.

• record_transformer

The record_transformer filter is a filter version of fluent-plugin-record-reformer output plugin.

<filter bar.**>
  @type record_transformer
  <record>
    hostname ${hostname}
  </record>
</filter>

Filters don't allow tag rewriting. So, unlike record_reformer, record_transformer doesn't have tag-related parameters.

Label

Label is useful for grouping Filter and Output Plugins. Labels simplify event stream routing and reduces the complexity of configuration files.

Again, seeing is believing:

# The events coming from this source are routed to the file <match>.
<source>
  @type forward
  port 24224
</source>

# The events from this source are routed to s3 <match> in <label @STAGING>
<source>
  @type forward
  @label @STAGING
  port 24225
</source>

<match forward.**>
  @type file
  # ...
</match>

# Label symbol should have `@` prefix bacause it is not a pattern.
<label @STAGING>
  <match forward.**>
    @type s3
    # ...
  </match>
</label>

At-Least-Once Semantics in Forward Plugins

Fluentd now supports at-least-once semantics between in_forward and out_forward. If you set require_ack_response parameter in out_forward plugin, out_forward waits ack response from in_forward.

<match forward.**>
  @type forward
  # ...
  require_ack_response true
  ack_response_timeout 60s # default is 190s
  <server>
    # ...
  </server>
</match>

Give it a Whirl

We tested v0.12 in some high-load environments without issues. That said, if you run into trouble with v0.12 in your environment, please file an issue on GitHub.

In addition, we are now updating Fluentd document for v0.12. More detailed information, e.g. how to write filter plugins, will be available soon!

Enjoy Fluentd!