Fluentd Blog

Fluentd v1.13.3 has been released

Hi users!

We have released v1.13.3. ChangeLog is here.

This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about in_tail bugs.

in_tail: Fixed a bug that DeletePending state is not cared on Windows

In this release, Windows specific bug was fixed for in_tail plugin.

Windows has a concept of DeletePending, which means "DeleteFile() was called on that file. NTFS will remove this file once everyone close the handlers for it".

In the previous versions, it didn't care about it correctly, so there was a case that deleted file could not be handled properly.

in_tail: Fixed a bug that detecting rotation twice on follow_inode

In this release, the bug that duplicate events are emitted when follow_inode is true and rotation is occurred.

This bug was occurred when the following conditions are met:

• Wrong inode is set to TailWatcher when follow_inode is true
• A key (TargetInfo) in @tails isn't updated for a same path even if new one has different inode

We recommend to upgrade if you enable follow_inode.

in_tail: Fixed some position file related bugs

In this release, some bugs which are related to position file was fixed.

Here is the detail of fixed bugs:

• When both follow_inodes and pos_file_compaction_interval are specified, file entries that didn't exist on start up will be deleted unexpectedly when compaction is triggered
• String#bytesize should be used instead of String#size for path length, but it is not used

It causes a bug that file compaction does not work as expected.

Upgrade attention for TD Agent users

If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because in_tail contains serious bugs in it.

Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.

$ sudo td-agent-gem install fluentd --version=1.13.3

Enjoy logging!

Read More

Fluentd v1.13.2 has been released

Hi users!

We have released v1.13.2. ChangeLog is here.

This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about crash bugs.

Fixed a crash bug during startup phase

In this release, a crash bug was fixed during outputting log at the early stage when parsing invalid config file.

This was a bug since v1.13.0. If you use invalid '@' prefix parameter, remove it as a workaround.

in_tail: Fixed a bug that remaining lines will be discarded

When rotation is occurred with the throttling feature is enabled, there is a case that remaining lines will be discarded.

Added support to customize configuration of oj options

In this release, it is able to customize OJ options via environment variables.

Use the following environment variables to customize behavior of OJ.

• FLUENT_OJ_OPTION_BIGDECIMAL_LOAD (float by default)
• FLUENT_OJ_OPTION_MAX_NESTING
• FLUENT_OJ_OPTION_MODE (compat by default)
• FLUENT_OJ_OPTION_USE_TO_JSON (true by default)

See Oj Options about details of oj options.

Generating template of storage plugin was supported

Since v1.13.2, fluent-plugin-generate command has been supported to generate template files for storage plugin.

$ fluent-plugin-generate storage foo
License: Apache-2.0
        create Gemfile
        create README.md
        create Rakefile
        create fluent-plugin-foo.gemspec
        create lib/fluent/plugin/storage_foo.rb
        create test/helper.rb
        create test/plugin/test_storage_foo.rb
Initialized empty Git repository in /work/fluentd/fluentd/fluentd/fluent-plugin-foo/.git/

Upgrade attention for TD Agent users

If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because in_tail contains serious bugs in it.

Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.

$ sudo td-agent-gem install fluentd --version=1.13.2

Enjoy logging!

Read More

Fluentd v1.13.1 has been released

Hi users!

We have released v1.13.1. ChangeLog is here.

This release is a maintenance release of v1.13 series.

in_tail: Fixed to remove too much verbose debugging logs

It was unexpectedly introduced by #3185 - log throttling feature in v1.13.0. In this release, too much verbose debug logging was removed.

It has nothing to do with it when you didn't use in_tail with debug level.

Upgrade attention for TD Agent users

If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4 or later. It is because in_tail contains serious bugs in it.

Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.

$ sudo td-agent-gem install fluentd --version=1.13.1

Enjoy logging!

Read More

Fluentd v1.13.0 has been released

Hi users!

We have released v1.13.0. ChangeLog is here.

There are some topics in this release.

• service_discovery: extended to support service discovery manager in simpler way
• in_tail: handling log throttling per file feature has been supported
• in_http: HTTP GET requests has been supported
• The log rotate settings in system configuration has been supported
• fluent-cat: the issue resending secondary file in specific format has been fixed

There are many in_tail bug fixes, we recommend to upgrade to v1.13.0.

service_discovery: extended to support service discovery manager in simpler way

In the previous versions, if plugin author supports service_discovery, it must build configurations structure explicitly, but it is complicated way.

Since Fluentd v1.13.0, it is enough to call #service_discovery_configure from Plugin#configure. It makes implementation costs lower.

in_tail: handling log throttling per file feature has been supported

In the previous versions, there is no way to limit log throttling.

Since Fluentd v1.13.0, the configuration parameter - read_bytes_limit_per_second is introduced.

<source>
  @type tail
  ...
  read_bytes_limit_per_second 4k
</source>

As read_bytes_limit_per_second accepts numbers with SI prefix such as 4k.

in_http: HTTP GET requests has been supported

This functionality is implemented to support health check feature on Azure App.

Azure App uses GET requests to check if the HTTP server is working all right. In the previous versions of Fluentd, it responds with "400 Bad Requests" to HTTP GET, so it does not work well for health checking purpose.

Since Fluentd v1.13.0, Fluentd returns a "200 OK".

Thanks Josh Keegan for requesting feature request!

The log rotate settings in system configuration has been supported

In the previous versions, log rotation options were supported as --log-rotate-age or --log-rotate-size via command line options.

On Windows, as Fluentd is launched as a windows service, it is required to configure again via --reg-winsvc-fluentdopt or edit fluentdopt registry key for log rotate customization.

These approach are not convenient for Windows users, so it may be better to support more comprehensive solution - customize via configuration file like this:

<system>
  <log>
    rotate_age 5
    rotate_size 1048576
  </log>
</system>

Now you can customize via rotate_age and rotate_size in log section.

fluent-cat: the issue resending secondary file in specific format has been fixed

The way to resend secondary is explained as How to resend secondary file, but it does not work if the record contains Fluent::EventTime. It raises Input must be a map (got Array) error.

In this release, it supports such a secondary file too.

Note that fluent-cat ignore timestamp which is stored in secondary records. (It is intended behavior for keeping consistency)

Miscellaneous bug fixes

• Fixed to disable trace_instruction when RubyVM::InstructionSequence is available. It improves compatibility with truffleruby some extent. #3376
• in_tail: Safely skip files which are used by another process on Windows. It improves exception handling about ERROR_SHARING_VIOLATION on Windows. #3378

Community topic

About three months has passed since community forum was moved to Discuss. Fluentd category is used actively nowadays! Note that fluentd forum on google groups is already deprecated.

Enjoy logging!

Read More

Fluentd v1.12.4 has been released

Hi users!

We have released v1.12.4. This release is a maintenance release of v1.12 series. ChangeLog is here.

in_tail: Fix a bug that refresh_watcher fails to handle file rotations

In the previous Fluentd v1.11.x implementation uses path based tailing keys. Since v1.12, follow_inodes feature is introduced, but as a side effects unexpectedly, refresh_watcher tend to fail to handle file rotations correctly.

This version contains a fix for the issue only. #3393

Enjoy logging!

Read More