Fluentd Blog

Fluentd v0.14.20 has been released

Hi users!

We have released v0.14.20. Here are the changes (full ChangeLog is available here): This release is mainly for bug fixes and new plugin helper.

We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.

record_accessor helper: New plugin helper for nested record access

We introduce new record_accessor helper. This provides unified nested record access mechanism for plugins. Some plugins don't support nested record access and it makes the configuration complicated. record_accessor helper mitigates this problem.

Here is an example of grep filter plugin with record_accessor:

<filter pattern>
  @type grep
  <regexp>
    key $.event.level # Access record["event"]["level"] field, not record["$.event.level"]
    pattern WARN
  </regexp>
</filter>

record_accessor supports two jsonpath like syntax:

• dot notation: $. started parameter. Chain fields by .

Simple syntax: $.event.level for record["event"]["level"], $.key1[0].key2 for record["key1"][0]["key2"]

• bracket notation: $[ started parameter. Chain fields by []

Useful for special characters, ., and etc: $["dot.key"][0]["space key"] for record["dot.key"][0]["space key"]

If you set non $. or $[ started value, e.g. key log, it is same as record["log"]. So using record_accessor doesn't break existing plugin behaviour.

To plugin developers, this is code example of plugin:

require 'fluent/plugin/filter'

module Fluent::Plugin
  class FooFilter < Filter
    Fluent::Plugin.register_filter('foo', self)

    # 1. load record_accessor helper
    helpers :record_accessor

    def configure(conf)
      # 2. Call `record_accessor_create` to create accessor object.
      # Actually, "$.key1.key2" comes from the configuration
      @accessor = record_accessor_create("$.key1.key2")
    end

    def filter(tag, time, record)
      # 3. Call `call` method
      value = @accessor.call(record)  # Get record["key1"]["key2"] field value

      # ...
    end
  end
end

Currently, filter_grep supports record_accessor. We will add record_accessor support to other plugins.

Fluentd log configuration: Add in directive

We sometimes got the request "We want fluentd's log as json format like Docker. json is easy to parse." This is good idea, so we add <log> directive to under <system> directive.

<system>
  <log>
    # text or json. default is text
    format json
    # Change format of log time. This affects both text and json.
    time_format FT%T%:z
  </log>
</system>

Here are log example:

• text

2017-07-27 07:37:02 +0900 [info]: #0 fluentd worker is now running worker=0

• json

{"time":"2017-07-27 06:44:54 +0900","level":"info","message":"fluentd worker is now running worker=0","worker_id":0}

Major bug fixes

• buf_file: Improve file handling to mitigate broken meta file

We got a bug report about broken meta file with buf_file. v0.14.20 includes the patch for this problem. We need the feedback the problem still happens or not.

• process: Send signal to all workers

Before, Supervisor sends a signal to latest one on multiple process environment. Supervisor now sends a signal to all workers.

Enjoy logging!