Fluentd Blog

Fluentd v1.8.0 has been released

Hi users!

We have released v1.8.0. ChangeLog is here. This release includes new service discovery plugin and lots of enhancements.

Add service discovery plugin

The service discovery plugin manages the list of destination by extensible way. Here is out_forward example with service discovery plugin:

# out_forward now uses service discovery plugin
<match test>
  @type forward
  <service_discovery>
    @type file
    path /path/to/sd_forward.yaml
  </service_discovery>
  # other parameters...
</match>

# content of sd_forward.yaml
- 'host': 127.0.0.1
  'port': 24224
  'weight': 1
  'name': server1
- 'host': 127.0.0.1
  'port': 24225
  'weight': 1
  'name': server2

With this configuration, out_forward uses file service discovery plugin to read the server list from sd_forward.yaml file.

In addition, file service discovery monitors path file. If sd_forward.yaml file is updated, the server list of out_forward is also updated. No need to restart fluentd for the server list of out_forward.

Supported built-in plugins are now static and file. We have a plan to support more service discovery plugins, e.g. DNS SRV record.

Improve config parameter handling

Current configuration parser has several problems:

• Automatic conversion for non string type

This is error prone. For example, if you set int_param "foo", int_param becomes 0. This case should raise configuration error.

• No way to use default value

This is critical when you use environment variable. We can't use nil or plugin's default value with param "#{ENV['FOO_PARAM']}" case.

For former case, we introduce --strict-config-value option for strict mode. With this option, fluentd raises configuration error for invalid parameter. You can also use <system> section instead of --strict-config-value.

<system>
  strict_config_value true
</system>

Fluentd v2 will use strict mode.

For latter case, we introduce helpers for ruby embedded parameter. You can use default value like below:

param "#{ENV['FOO_PARAM'] || use_default}"

If FOO_PARAM envvar doesn't exist, plugin's default will be used. This is useful on kubernetes environment.

formatter_csv: Support nested fields

You can use record accessor syntax for nested fields in fields parameter.

<format>
  @type csv
  fields code,message,$.nest.key
</format>

Deprecate top-level match for capturing fluentd logs

Fluentd now provides 2 approaches to capture fluentd logs.

• <label @FLUENT_LOG> with <filter> and <match>
• top level <match fluent.**>

Former is v1 style. Latter is for v0.12 compatibility. Fluentd now shows deprecation message for latter case.

Fluentd v2 will remove top level <match fluent.**> feature for fluentd logs.

Major bug fixes

• Fix error handling during dry-run mode. Show better logs
• server helper: Fix IPv6 dual stack mode issue for tcp socket
• server helper: Ignore ECONNREFUSED in TLS accept to avoid fluentd restart

Enjoy logging!