Fluentd Blog

Fluentd v0.14.9 has been released

Hi users!

We have just shipped Fluentd v0.14.9 including built-in plugin migration and bug fixes.

Here are major changes (full ChangeLog is available here):

We say again, fluentd v0.14 is still development version. If you try to use v0.14, check your configuration and plugins carefully.

Migrate several plugins to v0.14 API

We continue to migrate built-in plugins to v0.14 API. Here are migrated plugins in this release:

• in_http
• in_forward
• out_forward
• out_file
• out_exec
• out_exec_filter

We describe important changes of these plugins.

in_http

We removed un-documented detach process feature from in_http because DetachMultiProcessMixin module is deprecated in v0.14. If you set detach process related parameters in your configuration, it is ignored.

out_forward

Since this version, time_as_integer parameter is now false. It means v0.14.9's out_forward can't forward data to v0.12's in_forward by default. You need to set time_as_integer true explicitly. We have already mentioned this point in v0.14.0 release, so we hope this change doesn't break your v0.14 to v0.12 forwarding.

out_file

v0.14 Plugin API provides placeholder feature. You can emulate fluent-plugin-forest by more flexible way. Here is configuration example:

<match mydata>
  @type file
  path /path/to/${key}
  <buffer time,key>
    flush_at_shutdown true
  </buffer>
  <format>
    @type json
  </format>
</match>

${key} refers the key field of event record. If you pass {"key":"foo"} record, actual output file becomes /path/to/foo. The important point is if you want to refer time, tag or record keys on path, you need to list keys in <buffer CHUNK_KEYS> (See also v0.14 Plugin API slide). Here are several examples:

# Popular time, tag and key case
<match mydata>
  @type file
  path /path/to/%Y/%m/%d/${tag}/${key} # path is /path/to/2016/11/15/mydata/foo_0.log
  <buffer time,tag,key>
    flush_at_shutdown true
  </buffer>
  # ...
</match>

# Of course, you can use more keys
<match mydata>
  @type file
  path /path/to/${key1}/${key2}/{key3} # If record is {"key1":"foo","key2":"bar","key3":"baz"}, path is /path/to/foo/bar/baz.20161115_0.log
  <buffer time,key1,key2,key3>
    flush_at_shutdown true
  </buffer>
  # ...
</match>

out_file plugin requires time in CHUNK_KEYS because its placeholder is used in path implicitly.

We have a plan to migrate other 3rd party plugins to v0.14, e.g s3, kafka and more. We can say "Goodbye fluent-plugin-forest!" in near future.

Port Parser filter plugin

fluent-plugin-parser plugin is widely used in the world, so we decided to port this plugin into the core.

Note that we changed invalid event handling. fluent-plugin-parser logs warning when log is invalid. On the other hand, built-in parser filter emits invalid events to built-in @ERROR label. In this result, you can process invalid events using other plugins.

<source>
  @type forward
</source>

<filter app.**>
  @type parser
  key_name log
  <parse>
    @type json
  </parse>
</filter>

# If log field is json, record comes here
<match app.**>
  @type stdout
</match>

<label @ERROR>
  # If log field is not json, record comes here. Store such events into local file.
  <match app.**>
    @type file
    # ...
  </match>
</label>

record_transformer filter: Change default behaviours

record_transformer changes and removes old behaviours.

• auto_typecast is now true. It means the result of ${10 - 2} or ${record["int_field"]} is integer, not string, by default.
• Remove ${tags} placeholder. Use tag_parts instead.

We also have a plan to remove ${key} placeholder in the next version. Use ${record["key"]} instead.

Major bug fixes

• fluent-cat: Fix fluent-cat command to send sub-second precision time #1277
• out_forward: fix not to raise error when out_forward is initialized as secondary #1313

Enjoy logging!