Fluentd Blog

Fluentd v1.14.4 has been released

Hi users!

We have released v1.14.4. ChangeLog is here.

This release is a maintenance release of v1.14 series.

in_tail: Add max_line_size option to skip long lines

Starting from v1.14.4, in_tail supports a new option max_line_size that allows to skip lines above a certain size.

<source>
  @type tail
  path /var/log/nginx/*.log
  ...
  max_line_size 8MB  # skip very long lines
  ...
</source>

This option can be used to ensure that messages coming from in_tail is within chunk_limit_size, and will eliminate costly error handling on very large messages.

Improvement on BufferChunkOverflowError handling

Beginning with this release, Fluentd will no longer raise BufferChunkOverflowError on an oversized event stream if each event in the stream is smaller than chunk_limit_size

Example

es = Fluent::ArrayEventStream.new([[timestamp, {"message" => "a" * 1_000_000}],
                                 [timestamp, {"message" => "b" * 1_000_000}],
                                 [timestamp, {"message" => "c" * 1_000_000}]])

# Previously this could result in BufferChunkOverflowError even
# when chunk_limit_size is large enough (e.g. 1.2MB).
router.emit_stream(@tag, es)

See #3560 for more information.

out_file: Fix inccorect append writes on macOS

Prior to this release, out_file failed to write to files if append option was enabled on macOS with Ruby 2.7/3.0.

This issue was due to a bug in Ruby's IO implementation. This release contains a workaround for that bug.

See #3579 for details.

Enjoy logging!

Read More

Treasure Agent (td-agent) 3 will not be maintained anymore

Hi users and developers!

We announce the dropping schedule for TD Agent 3 development.

About Treasure Agent (td-agent) 3

Since td-agent 3.0.0 was released in 2017, td-agent 3 had been maintained for a long time.

As new major version of td-agent 4 was released in August 2020, and several updates was shipped, we decided to stop maintaining td-agent 3.

Scheduled end of life - Feb 2022

As you know, already stepping down maintenance activity, new minor update for td-agent 3 will not be shipped anymore.

Thus, we recommend to use td-agent 4, latest fluentd v1 series, for new deployment :)

How to migrate to Treasure Agent (td-agent) 4

There is a good article to do it.

• Upgrade td-agent from v3 to v4

Follow the above instructions.

Read More

td-agent v4.3.0 has been released

Hi users!

We have released td-agent v4.3.0. td-agent is a stable distribution package of Fluentd.

This release includes a security fix. Please check the topics below for details.

• CVE-2021-41186 - ReDoS vulnerability in parser_apache2.

Changes from td-agent v4.2.0

• Update bundled Fluentd from v1.13.3 to v1.14.3. Please see each release announcement of Fluentd for more detail:
  • Fluentd v1.14.0 has been released
  • Fluentd v1.14.1 has been released
  • Fluentd v1.14.2 has been released
  • Fluentd v1.14.3 has been released
• Update bundled Ruby from v2.7.4 to v2.7.5
• Support Debian 11 "bullseye"
• deb: Remove needless post-install script
• Windows: Add default log rotation setting

Please see ChangeLog for more detail.

Download

Please see the download page.

End of life announcement

• The package for Ubuntu xenial won't be provided from the next minor update (td-agent 4.4.x) because Ubuntu 16.04.x had already reached to End of Standard Support.
• td-agent 3 series won't be maintained anymore, we'll post another article for it.

Read More

Fluentd v1.14.3 has been released

Hi users!

We have released v1.14.3. ChangeLog is here.

This release is a maintenance release of v1.14 series.

in_tail: Fixed a bug that no new logs are read

This release contains a bug fix that no new logs were read when the following conditions were met.

• enable_stat_watcher is enabled. (default: true)
• enable_watch_timer is disabled. (default: true)

Thus, this bug was happened when you configure it like this:

<source>
  @type tail
  ...
  enable_stat_watcher true
  enable_watch_timer false
  ...
</source>

As enable_stat_watcher is enabled by default, this bug was occurred when enable_watch_timer false was explicitly set.

in_tail: Fixed a bug that the beginning and initial lines are lost

In this release, fixed a bug about in_tail that the beginning and initial lines are lost after startup when read_from_head false and path includes wildcard '*'.

This wrong behavior was caused by the following logic:

• New files which are detected on refresh are not read from the beginning and initial lines are lost
• Same as above for rotated files

Fixed a bug that processing messages were lost

In this release, fixed a bug that processing messages were lost when BufferChunkOverflowError was thrown even though only a specific message size exceeds chunk_limit_size.

Here is the example that it causes the bug:

  message a\n
  message b is longer.....message data exceeds chunk_limit_size\n
  message c\n

In above example, if incoming partial data exceeds chunk_limit_size, it raises BufferChunkOverflowError, then "message a" and "message c" is also lost.

It is not appropriate because the problem exists only a partial message data. Now such a message is changed to skip processing furthermore.

Note that even though with v1.14.3, still https://github.com/fluent/fluentd/issues/1849 is not fixed yet. This is also similar but another issue which cause BufferChunkOverflowError.

Upgrade attention for TD Agent users

If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because in_tail contains serious bugs in it.

As latest td-agent 4.2.0 bundles Fluentd 1.13.3. If you want to upgrade Fluentd further more, upgrade it by yourself.

$ sudo td-agent-gem install fluentd --version=1.14.3

Enjoy logging!

Read More

Fluentd v1.14.2 has been released

Hi users!

We have released v1.14.2. ChangeLog is here.

This release is a maintenance release of v1.14 series.

IMPORTANT: This release contain the fix for CVE-2021-41186 - ReDoS vulnerability in parser_apache2. This vulnerabbility is affected from Fluentd v0.14.14 to v1.14.1. We recommend to upgrade Fluentd to v1.14.2 or use patched version of parser_apache2 plugin.

Fixed ReDoS vulnerability in parser_apache2

parser_apache2 plugin in Fluentd v0.14.14 to v1.14.1 suffers from a regular expression denial of service (ReDoS) vulnerability.

A broken apache log with a certain pattern of string can spend too much time in a regular expression, resulting in the potential for a DoS attack.

There are two workarounds:

• Don't use parser_apache2 for parsing logs which cannot guarantee generated by Apache.
• Put patched version of parser_apache2.rb into /etc/fluent/plugin directory (or any other directories specified by the environment variable FLUENT_PLUGIN or --plugin option of fluentd).

As noted above, it requires a maliciously compromised Apache log to cause ReDoS. Usually, such a log will not be generated by Apache, so the risk of ReDoS is low.

Patched version of parser_apache2 is available from parser_apache2.rb

fluent-cat: Added --event-time option for testing

In this release, --event-time option was added to fluent-cat.

You can send event in specific event time with --event-time option instead of sending time in the previous versions.

echo '{"test": "ok"}' | fluent-cat --event-time "2021-10-29 13:14:15.0+00:00" tag

GitHub: Your Configuration field was automatically quoted

Recently, when creating issue on GitHub, Your Configuration field was automatically quoted.

Most users doesn't quote Your Configuration field manually when they report a bug. So it was changed to be quoted automatically.

Upgrade attention for TD Agent users

If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because in_tail contains serious bugs in it.

As latest td-agent 4.2.0 bundles Fluentd 1.13.3. If you want to upgrade Fluentd further more, upgrade it by yourself.

$ sudo td-agent-gem install fluentd --version=1.14.2

Enjoy logging!

Read More