Hi users!
We have released fluent-package v5.0.3. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)
This is a maintenance release of v5.0.x LTS series. As significant slow starting service and crash issues during startup on Windows were fixed, we recommend upgrading to fluent-package v5.0.3!
%~dp0 correctly.We plan to ship next LTS version of fluent-package v5.0.4 on June, 2024. The content of updates are still in T.B.D.
As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.
There is a exceptional maintenance release for v4.5.3 on Windows because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case.
We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5
Please see the download page.
Hi users!
We have released v1.16.5 on 2024-03-27. ChangeLog is here.
The previous version v1.16.4 has a serious Buffer bug when it processes large data exceeding chunk size limit. So, we have released v1.16.5 urgently, and have fixed the bug. So, please don't use v1.16.4.
Sorry for troubling.
In the previous version v1.16.4, we fixed a Buffer bug (Please see Fluentd v1.16.4 has been released for details).
There was a race condition problem with that fix, and similar Buffer errors explained in Fluentd v1.16.4 has been released could occur.
emit transaction failed: ...send an error event stream to @ERROR: ...These errors are the same as the Buffer error of Fluentd v1.16.4 has been released, but it would be more likely to happen. The cause is a race condition when processing large data exceeding chunk size limit. So, in that case, these errors can occur depending on the timing.
So, please don't use v1.16.4.
See #4447 for more information.
Sorry for troubling.
Hi users!
We have released v1.16.4 on 2024-03-14. ChangeLog is here.
In this release, we fixed several bugs and improved performance.
In previous versions, when Buffer processes large data exceeding the chunk size limit, it sometimes fails and causes an emit error.
It is usually unlikely, but it may occur when Buffer receives data with extremely large records that are distributed unevenly in the data.
If you set @ERROR label, the data is routed to that label, and Fluentd outputs a warning log message: send an error event stream to @ERROR: ....
However, if you don't set @ERROR label, the data is discarded, and Fluentd outputs a warning log message: emit transaction failed: ....
In addition, Input plugins that do not consider an emit error may stop working, such as in_windows_eventlog2.
Here is an example of the warning message:
2024-03-22 14:13:35 +0900 [warn]: #0 emit transaction failed: error_class=IOError error="closed stream" location="/path/to/fluentd/lib/fluent/plugin/buffer/file_chunk.rb:82:in `pos'" tag="test"
If you see these warnings, please update Fluentd.
See #4342 for more information.
in_tail: Fix tail watchers in rotate_wait state not being managed. #4334
Enjoy logging!
Hi users!
We have released fluent-package v5.0.2 and td-agent v4.5.2. fluent-package is a stable distribution package of Fluentd.
This is a maintenance release of v5.0.x LTS series.
As significant in_tail bugs (wrongly stopping tailing logs) were fixed in latest release, we recommend upgrading to fluent-package v5.0.2!
in_tail.
See Fluentd v1.16.3 and v1.16.2 have been released blog article about details.We plan to ship next LTS version of fluent-package v5.0.3 on Feb, 2024. The content of updates are still in T.B.D.
This is a exceptional maintenance release of v4.5.x series.
Fluentd was updated to 1.16.3 because it contains significant bug fixes about in_tail.
Note that td-agent will not be updated anymore.
See Drop schedule announcement about EOL of Treasure Agent (td-agent) 4.
Please see the download page.
Hi users!
We have released v1.16.3 (changelog) on 2023-11-14. We also describe the content of v1.16.2 (changelog) on 2023-07-14 here.
In these releases, in_tail issues that have been reported for years are fixed!
And other some bug fixes too.
in_tail: Fix in_tail wrongly stopping tailing the current target file and causing handle leaksThis issue has been reported for years. It was looking very serious, but the cause remained unclear until recently.
At last, we found different causes when the follow_inodes option is true and when it is false (default).
We fixed the problem of follow_inodes true case in v1.16.2, and fixed the problem of follow_inodes false case in v1.16.3.
Especially, in follow_inodes false case, the trouble was mainly caused by rotate_wait option.
In past versions, please avoid using large value for this option.
In particular, if you make it greater than refresh_interval option, it will surely cause this issue.
In Fluentd v1.15.1 (td-agent v4.4.0) or later, the following warning log is recorded when this issue occurs.
Skip update_watcher because watcher has been already updated by other inotify event
If this issue occurs, please restart Fluentd, and please update Fluentd to the latest version. fluent-package v5.0.2 will be released in this month and it contains these fixes.
See the following PRs for more information.
in_tail: Fix possible log duplication of follow_inodes option. #4237, #4239in_forward: Fix corrupted data possibly breaking other data. #4178NoMethodError. #4303Enjoy logging!
Hi users and developers!
We announce the dropping schedule for TD Agent 4 development.
Since td-agent 4.0.0 was released in 2020, td-agent 4 has been maintained for a long time.
As new major version of td-agent 4 was rebranded (to fluent-package v5), and released in August 2023,
we decided to stop maintaining td-agent 4.
As you know, already stepping down maintenance activity, new minor update for td-agent 4 will not be shipped anymore. (td-agent v4.5.1 will be the last release of this series, it will be shipped in this month.)
Thus, we recommend using fluent-package v5 for new deployment :)
There is a good article to do it.
Follow the above instructions.
Hi users!
We have released fluent-package v5.0.0 (Release Candidate) and
the following v5.0.1 (General Availability).
fluent-package is formerly known as td-agent and a stable distribution package of Fluentd.
In this blog article, we explains why the concept of channels is introduced.
In the previous versions, td-agent was shipped as quarterly releases in most cases.
It sometimes contains fluentd minor upgrade occasionally, which means that not only bug or security fixes,
but also contains new features. As a result, there was a case that the stability was affected unexpectedly.
In point view of adopting fluentd (and td-agent) for enterprise services, there was a requirement for the distribution which contains security or bug fixes only
and scheduled lifecycle of it.
The LTS release is the answer to support them.
There are two channels of support lifecycle.
Here is the difference of these channels.
fluent-package v6.0.0, and will be shipped in 2025.As for fluent-package v5,
Current releases and future plans:
| Release Version | Release Date | End of life |
|---|---|---|
| v4.5.1 | Aug, 2023 | Dec, 2023 |
| v5.0.1 (LTS) | Aug 29, 2023 | March, 2025 |
| v6.0.0 (LTS) | March, 2025? (T.B.D.) | T.B.D. |
Happy logging!
fluent-package "v5" is available since August 2023.
fluent-package is the successor of td-agent "v4".
In this post, we will share the steps we've tested and hopefully this will help your experience from v4 to v5.
fluent-package was formerly known as td-agent.
In the past, Treasure Data, Inc took the initiative to provide the package, but now the Fluentd community does it. This is why the package name was changed.
To represent "All in one package of Fluentd which contains Fluentd and related gem packages", the package name was changed to fluent-package.
Even though package name was changed, Treasure Data, Inc still sponsor the package delivery resources.
There are two channels for fluent-package.
One is normal release version which will be updated regularly - rapid release development style (td-agent v4 was released like this in almost every quarter).
In this version, fluentd will be eventually updated to newer minor version (e.g. 1.17.x and so on)
The other is a more conservative maintenance version (Long Term Support) which will not introduce a new feature. It only applies teeny update such as security or bug fix only. LTS for v5 will be supported until March, 2025.
More details about difference between normal release version and LTS version will be explained in Scheduled support lifecycle announcement about Fluent Package.
In the fluent-package v5, core components like ruby (2.7.8 -> 3.2.2) and OpenSSL (1.1.1 -> 3.1.0 for Windows, 3.0.8 for macOS) were updated.
The major changes are as follows.
td-agent command is renamed to fluentd.
$ td-agent --version -> $ fluentd --versiontd-agent-gem command is renamed to fluent-gem.
$ td-agent-gem list -> $ fluent-gem listtd-agent is renamed to fluentd.
$ systemctl status td-agent -> $ systemctl status fluentdWith the change of package name, install path, service name (e.g. /opt/fluent, fluentd.service) and so on were also changed.
Basically, for td-agent v4 users, it aims to keep compatibility as far as possible by executing the migration process with copying old files or providing
symbolic links for it.
If you want to know the details of upgraded components, see CHANGELOG.md.
NOTE: We explain for platform specific issue as "Additional hints for v4 users" section below.
During the upgrade process, plugins bundled in td-agent are automatically upgraded. With that being said, other plugins added on your own are not included. You should review if you need to upgrade plugins since some directory structures from v4 and v5 are changed.
In this post, I will show steps with plugins added on my own, "fluent-plugin-concat" for instance. Here is sample configuration file I used through steps.
<filter docker.log>
@type concat
key message
multiline_start_regexp /^Start/
</filter>
$ td-agent-gem list | grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-concat (2.5.0)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.0)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)
You can also find installed plugins under /opt/td-agent/lib/ruby/gems/2.7.0/gems/ directories.
$ ls -l /opt/td-agent/lib/ruby/gems/2.7.0/gems |grep fluent-plugin*
drwxr-xr-x. 5 root root 175 7月 14 03:01 fluent-plugin-calyptia-monitoring-0.1.3
drwxr-xr-x. 4 root root 206 7月 14 03:03 fluent-plugin-concat-2.5.0
drwxr-xr-x. 5 root root 4096 7月 14 03:01 fluent-plugin-elasticsearch-5.3.0
drwxr-xr-x. 4 root root 205 7月 14 03:01 fluent-plugin-flowcounter-simple-0.1.0
drwxr-xr-x. 6 root root 191 7月 14 03:01 fluent-plugin-kafka-0.19.0
drwxr-xr-x. 5 root root 190 7月 14 03:01 fluent-plugin-metrics-cmetrics-0.1.2
drwxr-xr-x. 5 root root 4096 7月 14 03:01 fluent-plugin-opensearch-1.1.0
drwxr-xr-x. 5 root root 215 7月 14 03:01 fluent-plugin-prometheus-2.0.3
drwxr-xr-x. 6 root root 238 7月 14 03:01 fluent-plugin-prometheus_pushgateway-0.1.0
drwxr-xr-x. 4 root root 176 7月 14 03:01 fluent-plugin-record-modifier-2.1.1
drwxr-xr-x. 3 root root 210 7月 14 03:01 fluent-plugin-rewrite-tag-filter-2.4.0
drwxr-xr-x. 5 root root 230 7月 14 03:01 fluent-plugin-s3-1.7.2
drwxr-xr-x. 3 root root 170 7月 14 03:01 fluent-plugin-sd-dns-0.1.0
drwxr-xr-x. 3 root root 49 7月 14 03:01 fluent-plugin-systemd-1.0.5
drwxr-xr-x. 5 root root 221 7月 14 03:01 fluent-plugin-td-1.2.0
drwxr-xr-x. 5 root root 166 7月 14 03:01 fluent-plugin-utmpx-0.5.0
drwxr-xr-x. 4 root root 191 7月 14 03:01 fluent-plugin-webhdfs-1.5.0
$ sudo systemctl stop td-agent
Even though fluent-package supports upgrade without stopping service, but recommend to stop explicitly.
When you use RedHat or derivative distributions, you can run following script if you want to install normal release version of fluent-package.
# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5.sh | sh
When you use RedHat or derivative distributions, you can run following script if you want to install LTS (Long term support) version of fluent-package.
# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5-lts.sh | sh
You can find more information about the installation script in Fluend Doc - Installation.
$ LANG=C yum info fluent-package
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: ftp.riken.jp
* extras: ftp.riken.jp
* updates: ftp.riken.jp
Installed Packages
Name : fluent-package
Arch : x86_64
Version : 5.0.0
Release : 1.el7
Size : 64 M
Repo : installed
From repo : /fluent-package-5.0.0-1.el7.x86_64
Summary : The stable distribution of Fluentd
URL : https://www.treasuredata.com/
License : ASL 2.0
Description : The stable distribution of Fluentd, called td-agent.
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now fluentd
$ fluent-gem list |grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.0)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)
You can see bundled plugins are upgraded as well but can not find plugins added on your own. In this post, added plugin was "fluent-plugin-concat" and it is not shown in installed list.
$ sudo fluent-gem install fluent-plugin-concat
$ fluent-gem list | grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-concat (2.5.0)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.0)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)
As for fluent-package v5, "fluent-plugin-concat" was installed under "/opt/fluent/lib/ruby/gems/3.2.0/gems/" directories.
$ ls -l /opt/fluent/lib/ruby/gems/3.2.0/gems/ |grep fluent-plugin*
drwxr-xr-x. 5 root root 175 7月 14 03:14 fluent-plugin-calyptia-monitoring-0.1.3
drwxr-xr-x. 4 root root 206 7月 14 03:16 fluent-plugin-concat-2.5.0
drwxr-xr-x. 5 root root 4096 7月 14 03:14 fluent-plugin-elasticsearch-5.3.0
drwxr-xr-x. 4 root root 205 7月 14 03:14 fluent-plugin-flowcounter-simple-0.1.0
drwxr-xr-x. 6 root root 191 7月 14 03:14 fluent-plugin-kafka-0.19.0
drwxr-xr-x. 5 root root 190 7月 14 03:14 fluent-plugin-metrics-cmetrics-0.1.2
drwxr-xr-x. 5 root root 4096 7月 14 03:14 fluent-plugin-opensearch-1.1.0
drwxr-xr-x. 5 root root 215 7月 14 03:14 fluent-plugin-prometheus-2.0.3
drwxr-xr-x. 6 root root 238 7月 14 03:14 fluent-plugin-prometheus_pushgateway-0.1.0
drwxr-xr-x. 4 root root 176 7月 14 03:14 fluent-plugin-record-modifier-2.1.1
drwxr-xr-x. 3 root root 210 7月 14 03:14 fluent-plugin-rewrite-tag-filter-2.4.0
drwxr-xr-x. 5 root root 230 7月 14 03:14 fluent-plugin-s3-1.7.2
drwxr-xr-x. 3 root root 170 7月 14 03:14 fluent-plugin-sd-dns-0.1.0
drwxr-xr-x. 3 root root 49 7月 14 03:14 fluent-plugin-systemd-1.0.5
drwxr-xr-x. 5 root root 221 7月 14 03:14 fluent-plugin-td-1.2.0
drwxr-xr-x. 5 root root 166 7月 14 03:14 fluent-plugin-utmpx-0.5.0
drwxr-xr-x. 4 root root 191 7月 14 03:14 fluent-plugin-webhdfs-1.5.0
$ sudo systemctl start fluentd
$ tail -100f /var/log/fluent/fluentd.log
Now, upgrading steps are completed. Happy Logging!
fluentd-apt-source package will be marked as a transitional package.
you can remove it safely with sudo apt purge fluentd-apt-source.td-agent.service, you must explicitly execute the following commands: $ sudo systemctl unmask td-agent
$ sudo systemctl enable fluentd
td-agent.service, you must explicitly execute the following command: $ sudo systemctl enable fluentd
fluent-package installer was changed not to start service by default.
If you want to start fluentd as a service, execute the following command with administrator privileges. c:\opt\fluent> net start fluentdwinsvc
WARNING: Currently we have no plan to officially support dmg version of fluent-package yet.
It is just modified to be a minimally buildable state, it is for testing purpose only.
Hi users!
We have released td-agent v4.5.0. td-agent is a stable distribution package of Fluentd.
We are preparing next major version of td-agent which bundles Ruby 3.2. It's still work in progerss, we'll announce it as soon as we are ready for it.
Please note that the package name will be changed to fluent-package.
Please see the following discussion for more detail: https://github.com/fluent/fluentd/discussions/3860
Please see the download page.
Hi users!
We have released v1.16.1 on 2023-04-17. ChangeLog is here.
In the previous version v1.16.0, there was a bug that caused Fluentd to fail to start with certain secondary configurations.
In this release, we fixed that problem and added a new feature of in_tcp.
And other some improvements!
in_tcp: Add message_length_limit to drop large incoming data. #4137out_secondary_file: Fix NameError of SecondaryFileOutput when setting other secondariesIn the previous version v1.16.0, there was a bug that if you set another secondary without ever setting out_secondary_file, Fluentd fails to start with the following error.
/path/to/fluentd/lib/fluent/plugin/output.rb:429:in `configure': uninitialized constant Fluent::Plugin::Output::SecondaryFileOutput (NameError)
if (@secondary.class != SecondaryFileOutput) &&
^^^^^^^^^^^^^^^^^^^
In this release, we fixed this problem.
See #4124 for more information.
Buffer: Fix that compress setting causes unexpected error when receiving already compressed MessagePackWith this release, it is now possible that Fluentd forwards already compressed MessagePack data to another Fluentd as is (without decompressing).
A possible use case is two-stage forwarding.
Forwarder1(out_forward) -> Forwarder2(in_forward, out_forward) -> Aggregator(in_forward)
In this case, Forwarder2 should process the data of CompressedMessagePackEventStream as is (without decompressing) and re-forward the data to Aggregator.
Until this version, a bug prevented us from setting Forwarder2's compress gzip of out_forward or buffer.
It resulted in the following error.
[error]: #0 unexpected error on reading data host="..." port=... error_class=ArgumentError error="unknown keyword: :packer"
So we had no choice but to decompress and forward it from Forwarder2 to Aggregator.
With this release, we can now set compress gzip for Forwarder2!
See #4146 and #4147 for more information.
UDPServer over max_bytes on Windows. #4131Enjoy logging!
Fluentd is an open source data collector to simplify log management.
2023-08-29: Drop schedule announcement about EOL of Treasure Agent (td-agent) 4
2023-08-29: Scheduled support lifecycle announcement about Fluent Package
2023-07-31: Upgrade to fluent-package v5
2024-03-29: fluent-package v5.0.3 has been released
2024-03-27: Fluentd v1.16.5 has been released
2024-03-14: Fluentd v1.16.4 has been released
2023-12-04: Open Source Summit Japan 2023
2023-11-29: fluent-package v5.0.2 has been released
2023-11-14: Fluentd v1.16.3 and v1.16.2 have been released
2023-08-29: Drop schedule announcement about EOL of Treasure Agent (td-agent) 4
2023-08-29: Scheduled support lifecycle announcement about Fluent Package
2023-07-31: Upgrade to fluent-package v5
2023-05-08: td-agent v4.5.0 has been released
Want to learn the basics of Fluentd? Check out these pages.
Couldn't find enough information? Let's ask the community!
You need commercial-grade support from Fluentd committers and experts?
©2010-2024 Fluentd Project. ALL Rights Reserved.
Fluentd is a hosted project under the Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.