Fluentd Blog

Fluentd v0.14.20 has been released

Hi users!

We have released v0.14.20. Here are the changes (full ChangeLog is available here): This release is mainly for bug fixes and new plugin helper.

We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.

record_accessor helper: New plugin helper for nested record access

We introduce new record_accessor helper. This provides unified nested record access mechanism for plugins. Some plugins don't support nested record access and it makes the configuration complicated. record_accessor helper mitigates this problem.

Here is an example of grep filter plugin with record_accessor:

<filter pattern>
  @type grep
  <regexp>
    key $.event.level # Access record["event"]["level"] field, not record["$.event.level"]
    pattern WARN
  </regexp>
</filter>

record_accessor supports two jsonpath like syntax:

• dot notation: $. started parameter. Chain fields by .

Simple syntax: $.event.level for record["event"]["level"], $.key1[0].key2 for record["key1"][0]["key2"]

• bracket notation: $[ started parameter. Chain fields by []

Useful for special characters, ., and etc: $["dot.key"][0]["space key"] for record["dot.key"][0]["space key"]

If you set non $. or $[ started value, e.g. key log, it is same as record["log"]. So using record_accessor doesn't break existing plugin behaviour.

To plugin developers, this is code example of plugin:

require 'fluent/plugin/filter'

module Fluent::Plugin
  class FooFilter < Filter
    Fluent::Plugin.register_filter('foo', self)

    # 1. load record_accessor helper
    helpers :record_accessor

    def configure(conf)
      # 2. Call `record_accessor_create` to create accessor object.
      # Actually, "$.key1.key2" comes from the configuration
      @accessor = record_accessor_create("$.key1.key2")
    end

    def filter(tag, time, record)
      # 3. Call `call` method
      value = @accessor.call(record)  # Get record["key1"]["key2"] field value

      # ...
    end
  end
end

Currently, filter_grep supports record_accessor. We will add record_accessor support to other plugins.

Fluentd log configuration: Add in directive

We sometimes got the request "We want fluentd's log as json format like Docker. json is easy to parse." This is good idea, so we add <log> directive to under <system> directive.

<system>
  <log>
    # text or json. default is text
    format json
    # Change format of log time. This affects both text and json.
    time_format FT%T%:z
  </log>
</system>

Here are log example:

• text

2017-07-27 07:37:02 +0900 [info]: #0 fluentd worker is now running worker=0

• json

{"time":"2017-07-27 06:44:54 +0900","level":"info","message":"fluentd worker is now running worker=0","worker_id":0}

Major bug fixes

• buf_file: Improve file handling to mitigate broken meta file

We got a bug report about broken meta file with buf_file. v0.14.20 includes the patch for this problem. We need the feedback the problem still happens or not.

• process: Send signal to all workers

Before, Supervisor sends a signal to latest one on multiple process environment. Supervisor now sends a signal to all workers.

Enjoy logging!

Read More

Fluentd v0.14.19 has been released

Hi users!

We have released v0.14.19. Here are the changes (full ChangeLog is available here): This release is mainly for bug fix and several plugin improvement.

We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.

filter_grep: New <regexp>/<exclude> section

We introduce <regexp>/<exclude> section to filter_grep. Here is an example:

<filter pattern>
  @type grep
  <regexp>
    key message
    pattern cool
  </regexp>
  <regexp>
    key hostname
    pattern ^web\d+\.example\.com$
  </regexp>
</filter>

This is same as:

<filter pattern>
  @type grep
  regexp1 message cool
  regexp2 hostname ^web\d+\.example\.com$
</filter>

New section approach is easy to parse by fluentd core and we have a plan to add nested record access support via new section parameters.

in_syslog improvement

• Allow more characters in identifier part

Old implementation allows only [a-zA-Z0-9_\/\.\-] characters in identifier part. This follows RFC3614 but real world systems use more characters. For the real world, in_syslog now accept more characters except , : and [.

• Add resolve_hostname parameter

This parameter is useful for reducing resolve cost when you don't use source_hostname_key.

Major bug fixes

• output: Fix race condition during retry

Users sometimes hit "undefined method `next_time' for nil:NilClass" error inside flush thread of output plugins. v0.14.19 fixed it.

We also released v0.12.38. This is for adding v0.14.19 features to in_syslog and filter_grep to reduce the differences between v0.12 and v0.14.

Enjoy logging!

Read More

Fluentd v0.14.17 has been released

Hi users!

We have released v0.14.17. Here are the changes (full ChangeLog is available here): This release is mainly for bug fixes.

We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.

server: Accept private key for TLS server without passphrase

The private_key_passphrase and ca_private_key_passphrase for TLS setting are now optional. It means you can use private key without passphrase in server plugin helper based plugins, e.g. in_forward/out_forward.

Major bug fixes

• out_forward: Fix ack handling
• config: Mask all secret parameters in worker section

This version fixed out_forward's require_ack_response regressions. If you use require_ack_response with v0.14, we recommend to update fluentd v0.14 to v0.14.17.

Enjoy logging!

Read More

Fluentd v0.14.16 has been released

Hi users!

We have released v0.14.16. Here are the changes (full ChangeLog is available here): This release is mainly for fixing regressions.

We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.

Allow null character in double-quoted string

You can use null character \0 with double-quoted string. This is useful for delimiter, e.g. tcp input, tsv parser, etc. Here is an example:

<source>
  @type tail
  path /path/to/app.log
  tag tail.test
  format tsv
  keys a,b,c
  delimiter "\0" # \0 or '\0' doesn't work
</source>

Major bug fixes

• out_forward: Call proper method for each connection type
• in_monitor_agent: check variable buffer is a Buffer instance

Enjoy logging!

Read More

Fluentd v0.14.15 has been released

Hi users!

We have released v0.14.15. Here are the changes (full ChangeLog is available here):

We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.

<worker N> directive

We introduced built-in multi-process feature since v0.14.12(See release announcement). New <worker N> directive enhances multi-process feature.

The limitation of multi-process was non-multi-process ready plugin can't be used together. For example, we can't use in_tail plugin with workers parameter. Following configuration raises an error at startup.

<system>
  workers 4
<system>
<source>
  @type forward
</source>
<source>
  @type tail
</source>

<worker N> directive resolves this problem. You can invoke any plugins in specific worker(N is worker index) and plugins under <worker N> acts as if they work on single process environment. We can mix multi-process ready and non-multi-process ready plugins in one instance. Working example with above configuration is like below:

<system>
  workers 4
<system>
# work on multi process environment. worker0 - worker3 invoke in_forward plugin
<source>
  @type forward
</source>
# work on only worker 0. worker1 - worker3 don't invoke in_tail plugin
<worker 0>
  <source>
    @type tail
  </source>
</worker>
# <worker 1>, <worker 2> or <worker 3> is also ok

Plugin: Add formatted_to_msgpack_binary? method to Output plugin API

We use formatted_to_msgpack_binary method to tell "This plugin uses msgpack for event serialization" to output core when you customize format method. This name is not rubyish, so we add formatted_to_msgpack_binary? method for better name. We recommend to use formatted_to_msgpack_binary? for new plugins.

Of course, formatted_to_msgpack_binary method is available for existing plugins.

Windows: Add several options to configure Fluentd windows service information

By default, Fluentd uses following settings for Windows service registration:

• Service name: fluentdwinsvc
• Display name: Fluentd Windows Service
• Service description: Fluentd is an event collector system.

Since v0.14.15, you can configure these settings via command line options.

• Service name: --winsvc-name NAME
• Display name: --winsvc-display-name DISPLAY_NAME
• Service description: --winsvc-desc DESC

Major bug fixes

• in_http: Fix X-Forwarded-For header handling. Accpet multiple headers
• windows: Quote the file path to the Ruby bin directory when starting fluentd as a windows service

Enjoy logging!

Read More