Fluentd Blog

td-agent v4.3.1 has been released

Hi users!

We have released td-agent v4.3.1. td-agent is a stable distribution package of Fluentd.

Changes from td-agent v4.3.0

• Update bundled Fluentd from v1.14.3 to v1.14.6. Please see each release announcement of Fluentd for more detail:
  • Fluentd v1.14.4 has been released
  • Fluentd v1.14.5 has been released
  • Fluentd v1.14.6 has been released
• Update bundled Ruby from v2.7.5 to v2.7.6
• Add some new plugins
  • fluent-plugin-opensearch
  • fluent-plugin-metrics-cmetrics
  • fluent-plugin-calyptia-monitoring
  • fluent-plugin-windows-exporter
• rpm: Remove .build-id directories

Please see ChangeLog for more detail.

Download

Please see the download page.

About packages for latest distributions

Although Ubuntu 22.04 LTS (Jammy Jellyfish) is already released, we don't yet provide the official td-agent package for it. An experimental package which bundles Ruby 3.1 is work in progress.

End of life announcement

• The package for Ubuntu xenial won't be provided from the next minor update (td-agent 4.4.0) because Ubuntu 16.04.x had already reached to End of Standard Support.
• td-agent 3 series won't be maintained anymore.

Read More

Fluentd v1.14.6 has been released

Hi users!

We have released v1.14.6. ChangeLog is here.

This release is a maintenance release of v1.14 series.

Enhancement

Enable server plugins to specify socket-option SO_LINGER

In this release, we add a new option linger_timeout to server plugin-helper so that we can specify SO_LINGER socket-option when using TCP or TLS server function of the helper.

The helper has used 0 for linger_timeout internally before. This made the connection closing immediate, and especially on non-Windows, this made the server send RST rather than FIN on the closing.

The main use of this option is to make the server send FIN on closing on non-Windows, by setting a positive value such as 1.

This new option is in transport section, and we can specify this for plugins using TCP or TLS server function of the helper, such as in_tcp, in_http, in_syslog, and in_forward.

<source>
  @type tcp
  ...
  <parse>
    ...
  </parse>
  <transport tcp>
    linger_timeout 1
  </transport>
</source>

Note: in_forward plugin had linger_timeout option before, but the option is now deprecated and unified to transport section.

See #3644 for more information.

Add --umask to fluentd command line parameter

When we launch Fluentd with --no-supervisor, the umask value of the environment is not applied because Fluentd overwrites the value.

In this release, We can use --umask option as a fluentd command line parameter to specify the umask value in the case of launching Fluentd with --no-supervisor.

$ fluentd -c {config_path} --no-supervisor --umask 002

See #3671, #3679 for more information.

Bug Fixes

Fix wrong calculation of retry count and interval

There were mistakes in the calculation about exponential_backoff retry_type.

In this release, the calculation is fixed, and the count and the interval change as following.

Count

• Before: At most retry_max_times + 1 retries were triggered.
• This Release: At most retry_max_times retries are triggered.

Interval

• Before: By default, n-th retry was triggered after 2^(n-1) seconds elapsed since the first time.
• This Release: By default, n-th retry is triggered after 2^(n-1) seconds elapsed since the last time.

When to switch to secondary

• Before: The total time calculation was wrong. Although retry_max_times + 1 retries were mistakenly triggered, retry_secondary_threshold was applied to the total time to the (retry_max_times)-th retry. So switching to secondary occurred earlier than the correct time.
• This Release: retry_secondary_threshold is correctly applied to the total retry time.

The examples of retry_max_times: 10 are as follows. (Set retry_randomize: false for testing.)

Only primary output

With secondary output

See #3649 for more information.

Apply modifications to the records being passed to @ERROR label

Filter plugins such as filter_record_transformer can modify the contents of records before the output.

If an error occurred in Output plugin such that records were passed to @ERROR label, those modifications were not applied to those records being passed.

In this release, records reflecting those modifications are now passed to @ERROR label.

See #3631 for more information.

Miscellaneous fixes

• Support IPv6 address for rpc_endpoint in system config. #3641.
• Fix some typo of metric names. #3630, #3673.
• Declare null_value_pattern as regexp. #3650.
• CI: Support Ruby 3.1 except Windows. #3619.
• Fix CHANGELOG.md heading styles. #3648.
• Switch to GitHub Discussions. #3654.

Enjoy logging!

Read More

Fluentd v1.14.5 has been released

Hi users!

We have released v1.14.5. ChangeLog is here.

This release is a maintenance release of v1.14 series.

in_http: Add support for "application/x-ndjson"

In this release, in_http supports the new Content-Type "application/x-ndjson". Its format is "ndjson", which is basically a list of JSON objects separated by "\n".

Here is an example of the format.

{"foo": "bar"}
{"buz": "hoge"}

See #3616 for more information.

Add support for the UCRT binary for Windows RubyInstaller 3.1

RubyInstaller 3.1 has switched C-Runtime from MSVCRT to UCRT, and the ruby and gem platform has changed to x64-mingw-ucrt. There was a problem in this environment that Windows-specific dependent gems could not be installed because Fluentd for this architecture had not been released.

Beginning with this release, Fluentd for this architecture is also released and this problem is solved.

See #3613 for more information.

out_forward: Fix hang-up issue during TLS handshake

There was a known issue that TLS handshake takes a long time or hangs when out_forward uses TLS transport, even though connect_timeout is specified.

With this release, connect_timeout is now reliably applied to TLS transport and able to prevent hangs.

See #3601 for more information.

Miscellaneous fixes

• Fix a bug of retrying once when retry_max_times is 0. #3608
• Fix "invalid byte sequence is replaced" warning not to write out invalid characters. #3596
• Bump up required ServerEngine to v2.2.5. #3599

Enjoy logging!

Read More

Fluentd v1.14.4 has been released

Hi users!

We have released v1.14.4. ChangeLog is here.

This release is a maintenance release of v1.14 series.

in_tail: Add max_line_size option to skip long lines

Starting from v1.14.4, in_tail supports a new option max_line_size that allows to skip lines above a certain size.

<source>
  @type tail
  path /var/log/nginx/*.log
  ...
  max_line_size 8MB  # skip very long lines
  ...
</source>

This option can be used to ensure that messages coming from in_tail is within chunk_limit_size, and will eliminate costly error handling on very large messages.

Improvement on BufferChunkOverflowError handling

Beginning with this release, Fluentd will no longer raise BufferChunkOverflowError on an oversized event stream if each event in the stream is smaller than chunk_limit_size

Example

es = Fluent::ArrayEventStream.new([[timestamp, {"message" => "a" * 1_000_000}],
                                 [timestamp, {"message" => "b" * 1_000_000}],
                                 [timestamp, {"message" => "c" * 1_000_000}]])

# Previously this could result in BufferChunkOverflowError even
# when chunk_limit_size is large enough (e.g. 1.2MB).
router.emit_stream(@tag, es)

See #3560 for more information.

out_file: Fix inccorect append writes on macOS

Prior to this release, out_file failed to write to files if append option was enabled on macOS with Ruby 2.7/3.0.

This issue was due to a bug in Ruby's IO implementation. This release contains a workaround for that bug.

See #3579 for details.

Enjoy logging!

Read More

Treasure Agent (td-agent) 3 will not be maintained anymore

Hi users and developers!

We announce the dropping schedule for TD Agent 3 development.

About Treasure Agent (td-agent) 3

Since td-agent 3.0.0 was released in 2017, td-agent 3 had been maintained for a long time.

As new major version of td-agent 4 was released in August 2020, and several updates was shipped, we decided to stop maintaining td-agent 3.

Scheduled end of life - Feb 2022

As you know, already stepping down maintenance activity, new minor update for td-agent 3 will not be shipped anymore.

Thus, we recommend to use td-agent 4, latest fluentd v1 series, for new deployment :)

How to migrate to Treasure Agent (td-agent) 4

There is a good article to do it.

• Upgrade td-agent from v3 to v4

Follow the above instructions.

Read More