Fluentd Blog

Fluentd v1.14.0 has been released

Hi users!

We have released v1.14.0. ChangeLog is here.

This release is a new release of v1.14 series. We recommend to upgrade Fluentd because it contains fixes about in_tail bugs because a bug that deleted paths are not removed was fixed.

System configuration about metrics parameters were added

In this release, new system configuration parameters were added.

• enable_input_metrics
• enable_size_metrics

You can enable these parameters in <system> directive.

<system>
  enable_input_metrics
  enable_size_metrics
</system>

This feature might need to pay higher CPU cost, so these event metrics features are disabled by default. These features are also enabled by --enable-input-metrics,--enable-size-metrics command line.

And more, metrics plugin interface was improved. This improvements are introduced to enhance collecting metrics functionality.

In the future, 3rd party plugin will be able to handle these metrics mechanism.

@ROOT label was changed as reserved label

In this release, the @ROOT label is reserved for a builtin label which is used for getting root router by plugin's event_emitter_router API.

This is incompatible change. Do not use @ROOT like <label @ROOT>.

This label is aimed to assign a label back to the default route. For example, timed-out event records are handled by the concat filter which can be sent to the default route.

in_syslog: send_keepalive_packet option was added

TCP keepalive was supported by in_syslog to avoid dead connection, as same as in_forward.

in_http: cors_allow_credentials option was added

This option tells browsers whether to expose the response to frontend when the credentials mode is include.

The default value is false.

Fixed a crash bug on logging peer host name with errors

When a client with invalid TLS certificates is trying to connect, it is hard to investigate which client has the problem without peer information, so it was changed to log peer information since v1.12.4.

But, if it fails to resolve peer name information when logging, it caused a crash bug unexpectedly. In this release, this bug was fixed.

Upgrade attention for TD Agent users

If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because in_tail contains serious bugs in it.

Note that the previous version of td-agent 4.1.1 ships a buggy Fluentd v1.12.3. We recommend upgrading to latest td-agent 4.2.0. It bundles Fluentd 1.13.3. If you want to upgrade Fluentd further more, upgrade it by yourself.

$ sudo td-agent-gem install fluentd --version=1.14.0

Enjoy logging!

Read More

TD Agent v4.2.0 has been released

Hi users!

We have released v4.2.0. ChangeLog is here.

This release is a maintenance release of v4.x series. We recommend to upgrade TD Agent because it contains latest Fluentd v1.13.3.

Jemalloc was downgraded to v3.6.0

Because of excessive amounts of memory consuming issue, bundled version of jemalloc was downgraded from v5.2.1 to v3.6.0.

This issue seems a well known issue as follows:

• https://bugs.ruby-lang.org/issues/14718
• https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html
• https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html

Jemalloc v3.6.0 is stable enough (according to jemlloc's web site, it still seems be supported)

See https://github.com/fluent-plugins-nursery/td-agent-builder/issues/305 about details.

Dependency to perl package was removed

In the previous versions, td-agent has dependency to perl packages. This is because it bundles jeprof, but users do not want to install extra dependencies. So, we decided to drop it.

Enjoy logging!

Read More

Fluentd v1.13.3 has been released

Hi users!

We have released v1.13.3. ChangeLog is here.

This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about in_tail bugs.

in_tail: Fixed a bug that DeletePending state is not cared on Windows

In this release, Windows specific bug was fixed for in_tail plugin.

Windows has a concept of DeletePending, which means "DeleteFile() was called on that file. NTFS will remove this file once everyone close the handlers for it".

In the previous versions, it didn't care about it correctly, so there was a case that deleted file could not be handled properly.

in_tail: Fixed a bug that detecting rotation twice on follow_inode

In this release, the bug that duplicate events are emitted when follow_inode is true and rotation is occurred.

This bug was occurred when the following conditions are met:

• Wrong inode is set to TailWatcher when follow_inode is true
• A key (TargetInfo) in @tails isn't updated for a same path even if new one has different inode

We recommend to upgrade if you enable follow_inode.

in_tail: Fixed some position file related bugs

In this release, some bugs which are related to position file was fixed.

Here is the detail of fixed bugs:

• When both follow_inodes and pos_file_compaction_interval are specified, file entries that didn't exist on start up will be deleted unexpectedly when compaction is triggered
• String#bytesize should be used instead of String#size for path length, but it is not used

It causes a bug that file compaction does not work as expected.

Upgrade attention for TD Agent users

If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because in_tail contains serious bugs in it.

Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.

$ sudo td-agent-gem install fluentd --version=1.13.3

Enjoy logging!

Read More

Fluentd v1.13.2 has been released

Hi users!

We have released v1.13.2. ChangeLog is here.

This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about crash bugs.

Fixed a crash bug during startup phase

In this release, a crash bug was fixed during outputting log at the early stage when parsing invalid config file.

This was a bug since v1.13.0. If you use invalid '@' prefix parameter, remove it as a workaround.

in_tail: Fixed a bug that remaining lines will be discarded

When rotation is occurred with the throttling feature is enabled, there is a case that remaining lines will be discarded.

Added support to customize configuration of oj options

In this release, it is able to customize OJ options via environment variables.

Use the following environment variables to customize behavior of OJ.

• FLUENT_OJ_OPTION_BIGDECIMAL_LOAD (float by default)
• FLUENT_OJ_OPTION_MAX_NESTING
• FLUENT_OJ_OPTION_MODE (compat by default)
• FLUENT_OJ_OPTION_USE_TO_JSON (true by default)

See Oj Options about details of oj options.

Generating template of storage plugin was supported

Since v1.13.2, fluent-plugin-generate command has been supported to generate template files for storage plugin.

$ fluent-plugin-generate storage foo
License: Apache-2.0
        create Gemfile
        create README.md
        create Rakefile
        create fluent-plugin-foo.gemspec
        create lib/fluent/plugin/storage_foo.rb
        create test/helper.rb
        create test/plugin/test_storage_foo.rb
Initialized empty Git repository in /work/fluentd/fluentd/fluentd/fluent-plugin-foo/.git/

Upgrade attention for TD Agent users

If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because in_tail contains serious bugs in it.

Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.

$ sudo td-agent-gem install fluentd --version=1.13.2

Enjoy logging!

Read More

Fluentd v1.13.1 has been released

Hi users!

We have released v1.13.1. ChangeLog is here.

This release is a maintenance release of v1.13 series.

in_tail: Fixed to remove too much verbose debugging logs

It was unexpectedly introduced by #3185 - log throttling feature in v1.13.0. In this release, too much verbose debug logging was removed.

It has nothing to do with it when you didn't use in_tail with debug level.

Upgrade attention for TD Agent users

If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4 or later. It is because in_tail contains serious bugs in it.

Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.

$ sudo td-agent-gem install fluentd --version=1.13.1

Enjoy logging!

Read More