td-agent v4.5.0 has been released

Hi users!

We have released td-agent v4.5.0. td-agent is a stable distribution package of Fluentd.

Changes from td-agent v4.4.2

  • Update bundled Ruby to 2.7.8 which is the last version of Ruby 2.7 series.
    • Only for RHEL 9 & Ubuntu 22.04 jammy, we updat Ruby to 3.1.4 exceptionally.
  • Update fluentd to 1.16.1
  • Update plugins
    • fluent-plugin-kafka 0.19.0
    • fluent-plugin-elasticsearch 5.3.0
    • fluent-plugin-opensearch 1.1.0
  • Drop Debian buster support

About next major version

We are preparing next major version of td-agent which bundles Ruby 3.2. It's still work in progerss, we'll announce it as soon as we are ready for it.

Please note that the package name will be changed to fluent-package. Please see the following discussion for more detail: https://github.com/fluent/fluentd/discussions/3860

Download

Please see the download page.

Read More

Fluentd v1.16.1 has been released

Hi users!

We have released v1.16.1 on 2023-04-17. ChangeLog is here.

In the previous version v1.16.0, there was a bug that caused Fluentd to fail to start with certain secondary configurations. In this release, we fixed that problem and added a new feature of in_tcp. And other some improvements!

Enhancement

  • in_tcp: Add message_length_limit to drop large incoming data. #4137

Bug Fixes

out_secondary_file: Fix NameError of SecondaryFileOutput when setting other secondaries

In the previous version v1.16.0, there was a bug that if you set another secondary without ever setting out_secondary_file, Fluentd fails to start with the following error.

/path/to/fluentd/lib/fluent/plugin/output.rb:429:in `configure': uninitialized constant Fluent::Plugin::Output::SecondaryFileOutput (NameError)

          if (@secondary.class != SecondaryFileOutput) &&
                                  ^^^^^^^^^^^^^^^^^^^

In this release, we fixed this problem.

See #4124 for more information.

Buffer: Fix that compress setting causes unexpected error when receiving already compressed MessagePack

With this release, it is now possible that Fluentd forwards already compressed MessagePack data to another Fluentd as is (without decompressing).

A possible use case is two-stage forwarding.

Forwarder1(out_forward) -> Forwarder2(in_forward, out_forward) -> Aggregator(in_forward)

In this case, Forwarder2 should process the data of CompressedMessagePackEventStream as is (without decompressing) and re-forward the data to Aggregator.

Until this version, a bug prevented us from setting Forwarder2's compress gzip of out_forward or buffer. It resulted in the following error.

[error]: #0 unexpected error on reading data host="..." port=... error_class=ArgumentError error="unknown keyword: :packer"

So we had no choice but to decompress and forward it from Forwarder2 to Aggregator.

With this release, we can now set compress gzip for Forwarder2!

See #4146 and #4147 for more information.

Others

  • Server helper: Suppress error of UDPServer over max_bytes on Windows. #4131

Enjoy logging!

Read More

Fluentd v1.16.0 has been released

Hi users!

We have released v1.16.0 on 2023-03-29. ChangeLog is here.

This release is a new release of v1.16 series. In this release, we enhanced the feature for chunk file corruption and fixed some bugs, mainly about logging and race condition errors.

Enhancement

Backup corrupted chunk files at resuming

Some chunk files in buffer_file or buffer_file_single are possibly corrupted after Fluentd stopped abnormally, such as due to a power failure.

Fluentd has a feature to backup unrecoverable chunks as before. However, when Fluentd resumes, remaining corrupted chunk files were just deleted.

Since this release, Fluentd also backups those files at resuming. In addition, Fluentd outputs some useful logs about possibly corrupted chunk files due to an abnormal termination.

Please see the document and the following PRs for more information.

in_tcp: Add send_keepalive_packet option

Since this release, you can use send_keepalive_packet option for in_tcp, as well as in_forward or in_syslog.

Please see the document and #3961 for more information.

Bug Fixes

  • out_secondary_file: Fix race condition. #4081
  • out_secondary_file: Suppress warning using different secondary for outsecondaryfile. #4087
  • out_forward: Fix error of ack handling conflict with require_ack_response. #4030
  • Windows: Fix bug that Fluentd sometimes tries to use an unavailable port and fails to start. #4092
  • Windows: Fix bug that flushing or graceful reloading cause reopening wrong log file. #4054
  • Fix not to log some initial messages without format setting applied. #4091
  • Fix bug that some system configs are not reflected. #4064, #4065
  • Non-Windows: Fix not to generate sigdump file after receiving SIGTERM. #4043
  • Fix bug that system_config.workers value can be wrongly 1 at config check at startup. #4066
  • Fix bug that a variable whose type is not Fluent::Config::Element can be passed to Fluent::Plugin::Base::configure(). #4066

Enjoy logging!

Read More

Fluentd v1.15.3 has been released

Hi users!

We have released v1.15.3 on 2022-11-02. ChangeLog is here.

In this release, we fixed several bugs.

Bug Fixes

out_file: Fix the multi-worker check with <worker N-M> directive

In the v1.15.1, we added a new feature to prevent write conflicts between multiple workers.

However, this fix didn't work correctly with <worker N-M> directive.

With this setting, the configuration check before starting the supervisor mistakenly determines that the setting of out_file is invalid and causes Fluentd to terminate.

In this release, we fixed this problem.

See #3942 for more information.

Fix log rotation on Windows

We can configure log rotation setting as follows.

On Windows, the log files must be separated by each process.

NOTE: When --log-rotate-age is specified on Windows, log files are separated into log-supervisor-0.log, log-0.log, ..., log-N.log where N is generation - 1 due to the system limitation. Windows does not permit delete and rename files simultaneously owned by another process.

However, that log rotation setting in the system-directive didn't satisfy this and Fluentd outputted the logs of the supervisor process into the log file of the worker-0 process.

This could cause errors during log rotation.

In this release, we fixed this problem.

See #3939 for more information.

Support glob for !include directive in YAML config format

In the v1.15.0, we supported YAML config format.

However, handling glob of !include directive was leaked.

In this release, we supported it.

See #3917 for more information.

Remove meaningless and unsecure oj options

In this release, we removed the following oj options.

  • FLUENT_OJ_OPTION_MODE=object
  • FLUENT_OJ_OPTION_MAX_NESTING

FLUENT_OJ_OPTION_MODE=object was unsecure, so we removed it.

This option is not used unless a user explicitly sets the environmental variable, so almost all users do not need to care about this vulnerability.

FLUENT_OJ_OPTION_MAX_NESTING was not supported, so we removed it.

See #3929 for more information.

Enjoy logging!

Read More

td-agent v4.4.2 has been released

Hi users!

We have released td-agent v4.4.2. td-agent is a stable distribution package of Fluentd.

Changes from td-agent v4.4.1

  • Update fluentd to 1.15.3
    • Fix log initializer to correctly create per-process files on Windows
    • out_file: Fix the multi-worker check with <worker 0-N> directive
    • Remove meaningless oj options
    • Support glob for !include directive in YAML config format
  • macOS: Support Apple Silicon
  • Update plugins
    • fluent-plugin-elasticsearch 5.2.4
    • fluent-plugin-opensearch 1.0.8
    • fluent-plugin-s3 1.7.2
    • fluent-plugin-windows-eventlog 0.8.2

Download

Please see the download page.

Read More

About Fluentd

Fluentd is an open source data collector to simplify log management.

Learn

Want to learn the basics of Fluentd? Check out these pages.

Ask the Community

Couldn't find enough information? Let's ask the community!

Ask the Experts

You need commercial-grade support from Fluentd committers and experts?

Follow Us!