Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More fluent-package v5.0.2 has been released https://www.fluentd.org/blog/fluent-package-v5.0.2-has-been-released <p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.2">v5.0.2</a> and td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.5.2">v4.5.2</a>. fluent-package is a stable distribution package of Fluentd. </p> <p>This is a maintenance release of v5.0.x LTS series. As significant <code>in_tail</code> bugs (wrongly stopping tailing logs) were fixed in latest release, we recommend upgrading to fluent-package v5.0.2!</p> <h3>Changes from fluent-package v5.0.1</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.3">1.16.3</a> which contains significant bug fixes about <code>in_tail</code>. See <a href="/blog/fluentd-v1.16.2-v1.16.3-have-been-released">Fluentd v1.16.3 and v1.16.2 have been released</a> blog article about details.</li> <li>Update plugins <ul> <li>fluent-diagtool v1.0.3. It supports fluent-package and can collect information about locally installed gems. It may help to migrate from td-agent v4 a bit. See <a href="/blog/upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a> for migration.</li> </ul></li> <li>msi: support path which contains space or parenthesis (<a href="https://github.com/fluent/fluent-package-builder/pull/589">#589</a>)</li> <li>deb: fixed system user/group name in logrotate config (<a href="https://github.com/fluent/fluent-package-builder/pull/592">#592</a>,<a href="https://github.com/fluent/fluent-package-builder/pull/594">#594</a>) <ul> <li>It fixes a bug that unknown user error was reported during log rotation.</li> </ul></li> <li>rpm: fixed to create fluentd user as system account (<a href="https://github.com/fluent/fluent-package-builder/pull/596">#596</a>) <ul> <li>It fixes a bug that /var/lib/fluent directory was created unexpectedly. It doesn&#39;t affect the fluentd service behavior, but it is desirable one.</li> </ul></li> <li>rpm: changed to keep system account after removing fluent-package. (<a href="https://github.com/fluent/fluent-package-builder/pull/598">#598</a>) <ul> <li>In the previous versions, there was a bug that group was not cleanly removed when the package was upgraded from td-agent v4. This change makes reinstall/downgrade friendly.</li> </ul></li> </ul> <h3>About next LTS schedule</h3> <p>We plan to ship next LTS version of fluent-package v5.0.3 on Feb, 2024. The content of updates are still in T.B.D.</p> <h3>About td-agent v4.5.2</h3> <p>This is a exceptional maintenance release of v4.5.x series. Fluentd was updated to 1.16.3 because it contains significant bug fixes about <code>in_tail</code>. Note that td-agent will not be updated anymore. See <a href="https://www.fluentd.org/blog/schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>.</p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> 2023-11-29 <p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.2">v5.0.2</a> and td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.5.2">v4.5.2</a>. fluent-package is a stable distribution package of Fluentd. </p> <p>This is a maintenance release of v5.0.x LTS series. As significant <code>in_tail</code> bugs (wrongly stopping tailing logs) were fixed in latest release, we recommend upgrading to fluent-package v5.0.2!</p> <h3>Changes from fluent-package v5.0.1</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.3">1.16.3</a> which contains significant bug fixes about <code>in_tail</code>. See <a href="/blog/fluentd-v1.16.2-v1.16.3-have-been-released">Fluentd v1.16.3 and v1.16.2 have been released</a> blog article about details.</li> <li>Update plugins <ul> <li>fluent-diagtool v1.0.3. It supports fluent-package and can collect information about locally installed gems. It may help to migrate from td-agent v4 a bit. See <a href="/blog/upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a> for migration.</li> </ul></li> <li>msi: support path which contains space or parenthesis (<a href="https://github.com/fluent/fluent-package-builder/pull/589">#589</a>)</li> <li>deb: fixed system user/group name in logrotate config (<a href="https://github.com/fluent/fluent-package-builder/pull/592">#592</a>,<a href="https://github.com/fluent/fluent-package-builder/pull/594">#594</a>) <ul> <li>It fixes a bug that unknown user error was reported during log rotation.</li> </ul></li> <li>rpm: fixed to create fluentd user as system account (<a href="https://github.com/fluent/fluent-package-builder/pull/596">#596</a>) <ul> <li>It fixes a bug that /var/lib/fluent directory was created unexpectedly. It doesn&#39;t affect the fluentd service behavior, but it is desirable one.</li> </ul></li> <li>rpm: changed to keep system account after removing fluent-package. (<a href="https://github.com/fluent/fluent-package-builder/pull/598">#598</a>) <ul> <li>In the previous versions, there was a bug that group was not cleanly removed when the package was upgraded from td-agent v4. This change makes reinstall/downgrade friendly.</li> </ul></li> </ul> <h3>About next LTS schedule</h3> <p>We plan to ship next LTS version of fluent-package v5.0.3 on Feb, 2024. The content of updates are still in T.B.D.</p> <h3>About td-agent v4.5.2</h3> <p>This is a exceptional maintenance release of v4.5.x series. Fluentd was updated to 1.16.3 because it contains significant bug fixes about <code>in_tail</code>. Note that td-agent will not be updated anymore. See <a href="https://www.fluentd.org/blog/schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>.</p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> Fluentd v1.16.3 and v1.16.2 have been released https://www.fluentd.org/blog/fluentd-v1.16.2-v1.16.3-have-been-released <p>Hi users!</p> <p>We have released v1.16.3 (<a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1163---20231114">changelog</a>) on 2023-11-14. We also describe the content of v1.16.2 (<a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1162---20230714">changelog</a>) on 2023-07-14 here.</p> <p>In these releases, <code>in_tail</code> issues that have been reported for years are fixed! And other some bug fixes too.</p> <h2>Bug Fixes</h2> <h3><code>in_tail</code>: Fix <code>in_tail</code> wrongly stopping tailing the current target file and causing handle leaks</h3> <p>This issue has been reported for years. It was looking very serious, but the cause remained unclear until recently.</p> <p>At last, we found different causes when the <a href="https://docs.fluentd.org/input/tail#follow_inodes">follow_inodes option</a> is <code>true</code> and when it is <code>false</code> (default). We fixed the problem of <code>follow_inodes true</code> case in v1.16.2, and fixed the problem of <code>follow_inodes false</code> case in v1.16.3.</p> <p>Especially, in <code>follow_inodes false</code> case, the trouble was mainly caused by <a href="https://docs.fluentd.org/input/tail#rotate_wait">rotate_wait option</a>. In past versions, please avoid using large value for this option. In particular, if you make it greater than <a href="https://docs.fluentd.org/input/tail#refresh_interval">refresh_interval option</a>, it will surely cause this issue.</p> <p>In Fluentd v1.15.1 (td-agent v4.4.0) or later, the following warning log is recorded when this issue occurs.</p> <pre><code>Skip update_watcher because watcher has been already updated by other inotify event </code></pre> <p>If this issue occurs, please restart Fluentd, and please update Fluentd to the latest version. fluent-package v5.0.2 will be released in this month and it contains these fixes.</p> <p>See the following PRs for more information.</p> <ul> <li><code>follow_inodes true</code> case: <a href="https://github.com/fluent/fluentd/pull/4208">#4208</a></li> <li><code>follow_inodes false</code> case: <a href="https://github.com/fluent/fluentd/pull/4327">#4327</a></li> </ul> <h3>Others</h3> <ul> <li><code>in_tail</code>: Fix possible log duplication of <a href="https://docs.fluentd.org/input/tail#follow_inodes">follow_inodes option</a>. <a href="https://github.com/fluent/fluentd/pull/4237">#4237</a>, <a href="https://github.com/fluent/fluentd/pull/4239">#4239</a></li> <li><code>in_forward</code>: Fix corrupted data possibly breaking other data. <a href="https://github.com/fluent/fluentd/pull/4178">#4178</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#ignore_same_log_interval">ignore_same_log_interval option</a>: Fix cache size growing unlimitedly. <a href="https://github.com/fluent/fluentd/pull/4229">#4229</a></li> <li><a href="https://docs.fluentd.org/deployment/trouble-shooting#dump-fluentds-internal-information">SIGDUMP</a>: Fix wrong value of object counts. <a href="https://github.com/fluent/fluentd/pull/4225">#4225</a></li> <li>Windows: Fix Fluentd failing to start when the log path isn&#39;t specified in the command line. <a href="https://github.com/fluent/fluentd/pull/4188">#4188</a></li> <li>Buffer: Fix <code>NoMethodError</code>. <a href="https://github.com/fluent/fluentd/pull/4303">#4303</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#rotate_age">rotate_age option</a>: Fix enum values causing setting error. <a href="https://github.com/fluent/fluentd/pull/4311">#4311</a></li> </ul> <p>Enjoy logging!</p> 2023-11-14 <p>Hi users!</p> <p>We have released v1.16.3 (<a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1163---20231114">changelog</a>) on 2023-11-14. We also describe the content of v1.16.2 (<a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1162---20230714">changelog</a>) on 2023-07-14 here.</p> <p>In these releases, <code>in_tail</code> issues that have been reported for years are fixed! And other some bug fixes too.</p> <h2>Bug Fixes</h2> <h3><code>in_tail</code>: Fix <code>in_tail</code> wrongly stopping tailing the current target file and causing handle leaks</h3> <p>This issue has been reported for years. It was looking very serious, but the cause remained unclear until recently.</p> <p>At last, we found different causes when the <a href="https://docs.fluentd.org/input/tail#follow_inodes">follow_inodes option</a> is <code>true</code> and when it is <code>false</code> (default). We fixed the problem of <code>follow_inodes true</code> case in v1.16.2, and fixed the problem of <code>follow_inodes false</code> case in v1.16.3.</p> <p>Especially, in <code>follow_inodes false</code> case, the trouble was mainly caused by <a href="https://docs.fluentd.org/input/tail#rotate_wait">rotate_wait option</a>. In past versions, please avoid using large value for this option. In particular, if you make it greater than <a href="https://docs.fluentd.org/input/tail#refresh_interval">refresh_interval option</a>, it will surely cause this issue.</p> <p>In Fluentd v1.15.1 (td-agent v4.4.0) or later, the following warning log is recorded when this issue occurs.</p> <pre><code>Skip update_watcher because watcher has been already updated by other inotify event </code></pre> <p>If this issue occurs, please restart Fluentd, and please update Fluentd to the latest version. fluent-package v5.0.2 will be released in this month and it contains these fixes.</p> <p>See the following PRs for more information.</p> <ul> <li><code>follow_inodes true</code> case: <a href="https://github.com/fluent/fluentd/pull/4208">#4208</a></li> <li><code>follow_inodes false</code> case: <a href="https://github.com/fluent/fluentd/pull/4327">#4327</a></li> </ul> <h3>Others</h3> <ul> <li><code>in_tail</code>: Fix possible log duplication of <a href="https://docs.fluentd.org/input/tail#follow_inodes">follow_inodes option</a>. <a href="https://github.com/fluent/fluentd/pull/4237">#4237</a>, <a href="https://github.com/fluent/fluentd/pull/4239">#4239</a></li> <li><code>in_forward</code>: Fix corrupted data possibly breaking other data. <a href="https://github.com/fluent/fluentd/pull/4178">#4178</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#ignore_same_log_interval">ignore_same_log_interval option</a>: Fix cache size growing unlimitedly. <a href="https://github.com/fluent/fluentd/pull/4229">#4229</a></li> <li><a href="https://docs.fluentd.org/deployment/trouble-shooting#dump-fluentds-internal-information">SIGDUMP</a>: Fix wrong value of object counts. <a href="https://github.com/fluent/fluentd/pull/4225">#4225</a></li> <li>Windows: Fix Fluentd failing to start when the log path isn&#39;t specified in the command line. <a href="https://github.com/fluent/fluentd/pull/4188">#4188</a></li> <li>Buffer: Fix <code>NoMethodError</code>. <a href="https://github.com/fluent/fluentd/pull/4303">#4303</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#rotate_age">rotate_age option</a>: Fix enum values causing setting error. <a href="https://github.com/fluent/fluentd/pull/4311">#4311</a></li> </ul> <p>Enjoy logging!</p> Drop schedule announcement about EOL of Treasure Agent (td-agent) 4 https://www.fluentd.org/blog/schedule-for-td-agent-4-eol <p>Hi users and developers!</p> <p>We announce the dropping schedule for TD Agent 4 development.</p> <h2>About Treasure Agent (td-agent) 4</h2> <p>Since td-agent 4.0.0 was released in 2020, td-agent 4 has been maintained for a long time.</p> <p>As new major version of td-agent 4 was rebranded (to <code>fluent-package</code> v5), and released in August 2023, we decided to stop maintaining td-agent 4.</p> <h2>Scheduled end of life - Dec 2023</h2> <p>As you know, already stepping down maintenance activity, new minor update for td-agent 4 will not be shipped anymore. (td-agent v4.5.1 will be the last release of this series, it will be shipped in this month.)</p> <p>Thus, we recommend using fluent-package v5 for new deployment :)</p> <h2>How to migrate to Fluent Package v5</h2> <p>There is a good article to do it.</p> <ul> <li><a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></li> </ul> <p>Follow the above instructions.</p> 2023-08-29 <p>Hi users and developers!</p> <p>We announce the dropping schedule for TD Agent 4 development.</p> <h2>About Treasure Agent (td-agent) 4</h2> <p>Since td-agent 4.0.0 was released in 2020, td-agent 4 has been maintained for a long time.</p> <p>As new major version of td-agent 4 was rebranded (to <code>fluent-package</code> v5), and released in August 2023, we decided to stop maintaining td-agent 4.</p> <h2>Scheduled end of life - Dec 2023</h2> <p>As you know, already stepping down maintenance activity, new minor update for td-agent 4 will not be shipped anymore. (td-agent v4.5.1 will be the last release of this series, it will be shipped in this month.)</p> <p>Thus, we recommend using fluent-package v5 for new deployment :)</p> <h2>How to migrate to Fluent Package v5</h2> <p>There is a good article to do it.</p> <ul> <li><a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></li> </ul> <p>Follow the above instructions.</p> Scheduled support lifecycle announcement about Fluent Package https://www.fluentd.org/blog/fluent-package-scheduled-lifecycle <p>Hi users!</p> <p>We have released <code>fluent-package</code> <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.0">v5.0.0</a> (Release Candidate) and the following <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.1">v5.0.1</a> (General Availability). fluent-package is formerly known as <code>td-agent</code> and a stable distribution package of Fluentd.</p> <p>In this blog article, we explains why the concept of channels is introduced.</p> <h3>Why is LTS (Long Term Support) channel introduced?</h3> <p>In the previous versions, <code>td-agent</code> was shipped as quarterly releases in most cases. It sometimes contains fluentd minor upgrade occasionally, which means that not only bug or security fixes, but also contains new features. As a result, there was a case that the stability was affected unexpectedly.</p> <p>In point view of adopting fluentd (and <code>td-agent</code>) for enterprise services, there was a requirement for the distribution which contains security or bug fixes only and scheduled lifecycle of it.</p> <p>The LTS release is the answer to support them.</p> <h3>Channels for updates</h3> <p>There are two channels of support lifecycle.</p> <ul> <li>Normal release</li> <li>LTS (Long Term Support)</li> </ul> <p>Here is the difference of these channels.</p> <ul> <li>Normal release (5.x series) <ul> <li>Recommended for early adopters (eager to try new features of Fluentd)</li> <li>Rapid release development (planned semi-annually releases, but it may vary.)</li> <li>Fluentd will be upgraded occasionally <ul> <li>Minor upgrade will be applied. e.g. 1.16 =&gt; 1.17 and so on.</li> <li>e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), ... v5.1.0 (Fluentd v1.17.0)</li> </ul></li> <li>The only latest version will be supported (bug fix will not be backported into older version)</li> </ul></li> <li>LTS (Long Term Support, 5.0.x series) <ul> <li>Recommended for enterprise services</li> <li>Security and bug fix release only</li> <li>Fluentd will be upgraded only in the teeny version and will stick to 1.16.x series. <ul> <li>Thus, the version number will be 5.0.x. (5.1 series will not be shipped)</li> <li>e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), v5.0.3 (Fluentd v1.16.4) ... v5.0.x (Fluentd v1.16.x)</li> </ul></li> <li>Next major version will be <code>fluent-package</code> v6.0.0, and will be shipped in 2025.</li> </ul></li> </ul> <p>As for fluent-package v5,</p> <ul> <li>fluent-package v5 LTS will be supported until March, 2025 (Even though Ruby 3.2 is still supported)</li> <li>Until 5.1.0 is released, normal release channel and LTS channel are the same package.</li> </ul> <p>Current releases and future plans:</p> <p><img src="/images/blog/20230829_fluent-package-scheduled-lifecycle.png" alt=""></p> <!-- Generated by mermaid-js: npx mmdc -t default -i fluent-package-scheduled-lifecycle.mmd -o fluent-package-scheduled-lifecycle.png gantt title Scheduled Support lifecycle for Fluent Package dateFormat YYYY-MM axisFormat %Y-%m todayMarker off section v4 %% date -d '20230508 113 days' +%Y%m%d %% => 20230829 v4.5.0 :done, v450, 2023-05-08, 113d %% date -d '20230829 124 days' +%Y%m%d %% => 20231231 v4.5.1 :active, v451, after v450, 124d section v5.x %% date -d '20230729 611 days' +%Y%m%d %% => 202350331 v5.0.0 :active, v500, 2023-07-29, 30d v5.0.1 (T.B.D.):v501, after v500, 120d v5.0.x (fluentd 1.16.x T.B.D.):v50x, after v501, 230d v5.1.x (fluentd 1.17.x T.B.D.):v51x, after v50x, 231d section v5.0 (LTS) v5.0.x (LTS) :lts_v5, 2023-07-29, 611d Fluent Pacakge v6.0.x (LTS) :milestone, lts_v6, 2025-03-31, 10d --> <table border=1 width="100%"> <tr> <th style='background-color: #eee'>Release Version</th> <th style='background-color: #eee'>Release Date</th> <th style='background-color: #eee'>End of life</th> </tr> <tr> <td>v4.5.1</td> <td>Aug, 2023</td> <td>Dec, 2023</td> </tr> <tr> <td>v5.0.1 (LTS)</td> <td>Aug 29, 2023</td> <td>March, 2025</td> </tr> <tr> <td>v6.0.0 (LTS)</td> <td>March, 2025? (T.B.D.)</td> <td>T.B.D.</td> </tr> </table> <p>Happy logging!</p> 2023-08-29 <p>Hi users!</p> <p>We have released <code>fluent-package</code> <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.0">v5.0.0</a> (Release Candidate) and the following <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.1">v5.0.1</a> (General Availability). fluent-package is formerly known as <code>td-agent</code> and a stable distribution package of Fluentd.</p> <p>In this blog article, we explains why the concept of channels is introduced.</p> <h3>Why is LTS (Long Term Support) channel introduced?</h3> <p>In the previous versions, <code>td-agent</code> was shipped as quarterly releases in most cases. It sometimes contains fluentd minor upgrade occasionally, which means that not only bug or security fixes, but also contains new features. As a result, there was a case that the stability was affected unexpectedly.</p> <p>In point view of adopting fluentd (and <code>td-agent</code>) for enterprise services, there was a requirement for the distribution which contains security or bug fixes only and scheduled lifecycle of it.</p> <p>The LTS release is the answer to support them.</p> <h3>Channels for updates</h3> <p>There are two channels of support lifecycle.</p> <ul> <li>Normal release</li> <li>LTS (Long Term Support)</li> </ul> <p>Here is the difference of these channels.</p> <ul> <li>Normal release (5.x series) <ul> <li>Recommended for early adopters (eager to try new features of Fluentd)</li> <li>Rapid release development (planned semi-annually releases, but it may vary.)</li> <li>Fluentd will be upgraded occasionally <ul> <li>Minor upgrade will be applied. e.g. 1.16 =&gt; 1.17 and so on.</li> <li>e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), ... v5.1.0 (Fluentd v1.17.0)</li> </ul></li> <li>The only latest version will be supported (bug fix will not be backported into older version)</li> </ul></li> <li>LTS (Long Term Support, 5.0.x series) <ul> <li>Recommended for enterprise services</li> <li>Security and bug fix release only</li> <li>Fluentd will be upgraded only in the teeny version and will stick to 1.16.x series. <ul> <li>Thus, the version number will be 5.0.x. (5.1 series will not be shipped)</li> <li>e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), v5.0.3 (Fluentd v1.16.4) ... v5.0.x (Fluentd v1.16.x)</li> </ul></li> <li>Next major version will be <code>fluent-package</code> v6.0.0, and will be shipped in 2025.</li> </ul></li> </ul> <p>As for fluent-package v5,</p> <ul> <li>fluent-package v5 LTS will be supported until March, 2025 (Even though Ruby 3.2 is still supported)</li> <li>Until 5.1.0 is released, normal release channel and LTS channel are the same package.</li> </ul> <p>Current releases and future plans:</p> <p><img src="/images/blog/20230829_fluent-package-scheduled-lifecycle.png" alt=""></p> <!-- Generated by mermaid-js: npx mmdc -t default -i fluent-package-scheduled-lifecycle.mmd -o fluent-package-scheduled-lifecycle.png gantt title Scheduled Support lifecycle for Fluent Package dateFormat YYYY-MM axisFormat %Y-%m todayMarker off section v4 %% date -d '20230508 113 days' +%Y%m%d %% => 20230829 v4.5.0 :done, v450, 2023-05-08, 113d %% date -d '20230829 124 days' +%Y%m%d %% => 20231231 v4.5.1 :active, v451, after v450, 124d section v5.x %% date -d '20230729 611 days' +%Y%m%d %% => 202350331 v5.0.0 :active, v500, 2023-07-29, 30d v5.0.1 (T.B.D.):v501, after v500, 120d v5.0.x (fluentd 1.16.x T.B.D.):v50x, after v501, 230d v5.1.x (fluentd 1.17.x T.B.D.):v51x, after v50x, 231d section v5.0 (LTS) v5.0.x (LTS) :lts_v5, 2023-07-29, 611d Fluent Pacakge v6.0.x (LTS) :milestone, lts_v6, 2025-03-31, 10d --> <table border=1 width="100%"> <tr> <th style='background-color: #eee'>Release Version</th> <th style='background-color: #eee'>Release Date</th> <th style='background-color: #eee'>End of life</th> </tr> <tr> <td>v4.5.1</td> <td>Aug, 2023</td> <td>Dec, 2023</td> </tr> <tr> <td>v5.0.1 (LTS)</td> <td>Aug 29, 2023</td> <td>March, 2025</td> </tr> <tr> <td>v6.0.0 (LTS)</td> <td>March, 2025? (T.B.D.)</td> <td>T.B.D.</td> </tr> </table> <p>Happy logging!</p> Upgrade to fluent-package v5 https://www.fluentd.org/blog/upgrade-td-agent-v4-to-v5 <p><code>fluent-package</code> &quot;v5&quot; is available since August 2023. <code>fluent-package</code> is the successor of <code>td-agent</code> &quot;v4&quot;.</p> <p>In this post, we will share the steps we&#39;ve tested and hopefully this will help your experience from v4 to v5.</p> <h2>Why package was renamed?</h2> <p><code>fluent-package</code> was formerly known as <code>td-agent</code>. In the past, <a href="http://www.treasuredata.com/">Treasure Data, Inc</a> took the initiative to provide the package, but now the Fluentd community does it. This is why the package name was changed.</p> <p>To represent &quot;All in one package of Fluentd which contains Fluentd and related gem packages&quot;, the package name was changed to <code>fluent-package</code>.</p> <p>Even though package name was changed, Treasure Data, Inc still sponsor the package delivery resources.</p> <h2>Which channel should I use?</h2> <p>There are two channels for <code>fluent-package</code>.</p> <ul> <li>Normal release version</li> <li>Long Term Support version</li> </ul> <p>One is normal release version which will be updated regularly - rapid release development style (<code>td-agent v4</code> was released like this in almost every quarter). In this version, fluentd will be eventually updated to newer minor version (e.g. 1.17.x and so on)</p> <p>The other is a more conservative maintenance version (Long Term Support) which will not introduce a new feature. It only applies teeny update such as security or bug fix only. LTS for v5 will be supported until March, 2025.</p> <p>More details about difference between normal release version and LTS version will be explained in <a href="fluent-package-scheduled-lifecycle">Scheduled support lifecycle announcement about Fluent Package</a>.</p> <h2>Differences between td-agent v4 and fluent-package v5</h2> <p>In the <code>fluent-package</code> v5, core components like ruby (2.7.8 -&gt; 3.2.2) and OpenSSL (1.1.1 -&gt; 3.1.0 for Windows, 3.0.8 for macOS) were updated.</p> <p>The major changes are as follows.</p> <ul> <li><code>td-agent</code> command is renamed to <code>fluentd</code>. <ul> <li><code>$ td-agent --version</code> -&gt; <code>$ fluentd --version</code></li> </ul></li> <li><code>td-agent-gem</code> command is renamed to <code>fluent-gem</code>. <ul> <li><code>$ td-agent-gem list</code> -&gt; <code>$ fluent-gem list</code></li> </ul></li> <li>The service name for non-Windows <code>td-agent</code> is renamed to <code>fluentd</code>. <ul> <li><code>$ systemctl status td-agent</code> -&gt; <code>$ systemctl status fluentd</code></li> </ul></li> </ul> <p>With the change of package name, install path, service name (e.g. /opt/fluent, fluentd.service) and so on were also changed. Basically, for <code>td-agent</code> v4 users, it aims to keep compatibility as far as possible by executing the migration process with copying old files or providing symbolic links for it.</p> <p>If you want to know the details of upgraded components, see <a href="https://github.com/fluent/fluent-package-builder/blob/master/CHANGELOG.md#release-v500---20230729">CHANGELOG.md</a>.</p> <p>NOTE: We explain for platform specific issue as &quot;Additional hints for v4 users&quot; section below.</p> <h2>Upgrade steps</h2> <p>During the upgrade process, plugins bundled in <code>td-agent</code> are automatically upgraded. With that being said, other plugins added on your own are not included. You should review if you need to upgrade plugins since some directory structures from v4 and v5 are changed.</p> <p>In this post, I will show steps with plugins added on my own, <strong>&quot;fluent-plugin-concat&quot;</strong> for instance. Here is sample configuration file I used through steps.</p> <pre><code>&lt;filter docker.log&gt; @type concat key message multiline_start_regexp /^Start/ &lt;/filter&gt; </code></pre> <h3>1. Review what plugins are installed together with td-agent v4.</h3> <pre><code>$ td-agent-gem list | grep fluent-plugin* fluent-plugin-calyptia-monitoring (0.1.3) fluent-plugin-concat (2.5.0) fluent-plugin-elasticsearch (5.3.0) fluent-plugin-flowcounter-simple (0.1.0) fluent-plugin-kafka (0.19.0) fluent-plugin-metrics-cmetrics (0.1.2) fluent-plugin-opensearch (1.1.0) fluent-plugin-prometheus (2.0.3) fluent-plugin-prometheus_pushgateway (0.1.0) fluent-plugin-record-modifier (2.1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.2) fluent-plugin-sd-dns (0.1.0) fluent-plugin-systemd (1.0.5) fluent-plugin-td (1.2.0) fluent-plugin-utmpx (0.5.0) fluent-plugin-webhdfs (1.5.0) </code></pre> <p>You can also find installed plugins under <code>/opt/td-agent/lib/ruby/gems/2.7.0/gems/</code> directories.</p> <pre><code>$ ls -l /opt/td-agent/lib/ruby/gems/2.7.0/gems |grep fluent-plugin* drwxr-xr-x. 5 root root 175 7月 14 03:01 fluent-plugin-calyptia-monitoring-0.1.3 drwxr-xr-x. 4 root root 206 7月 14 03:03 fluent-plugin-concat-2.5.0 drwxr-xr-x. 5 root root 4096 7月 14 03:01 fluent-plugin-elasticsearch-5.3.0 drwxr-xr-x. 4 root root 205 7月 14 03:01 fluent-plugin-flowcounter-simple-0.1.0 drwxr-xr-x. 6 root root 191 7月 14 03:01 fluent-plugin-kafka-0.19.0 drwxr-xr-x. 5 root root 190 7月 14 03:01 fluent-plugin-metrics-cmetrics-0.1.2 drwxr-xr-x. 5 root root 4096 7月 14 03:01 fluent-plugin-opensearch-1.1.0 drwxr-xr-x. 5 root root 215 7月 14 03:01 fluent-plugin-prometheus-2.0.3 drwxr-xr-x. 6 root root 238 7月 14 03:01 fluent-plugin-prometheus_pushgateway-0.1.0 drwxr-xr-x. 4 root root 176 7月 14 03:01 fluent-plugin-record-modifier-2.1.1 drwxr-xr-x. 3 root root 210 7月 14 03:01 fluent-plugin-rewrite-tag-filter-2.4.0 drwxr-xr-x. 5 root root 230 7月 14 03:01 fluent-plugin-s3-1.7.2 drwxr-xr-x. 3 root root 170 7月 14 03:01 fluent-plugin-sd-dns-0.1.0 drwxr-xr-x. 3 root root 49 7月 14 03:01 fluent-plugin-systemd-1.0.5 drwxr-xr-x. 5 root root 221 7月 14 03:01 fluent-plugin-td-1.2.0 drwxr-xr-x. 5 root root 166 7月 14 03:01 fluent-plugin-utmpx-0.5.0 drwxr-xr-x. 4 root root 191 7月 14 03:01 fluent-plugin-webhdfs-1.5.0 </code></pre> <h3>2. Stop td-agent v4 daemon.</h3> <pre><code>$ sudo systemctl stop td-agent </code></pre> <p>Even though <code>fluent-package</code> supports upgrade without stopping service, but recommend to stop explicitly.</p> <h3>3. Run installation script of fluent-package v5.</h3> <p>When you use RedHat or derivative distributions, you can run following script if you want to install normal release version of <code>fluent-package</code>.</p> <pre><code># curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5.sh | sh </code></pre> <p>When you use RedHat or derivative distributions, you can run following script if you want to install LTS (Long term support) version of <code>fluent-package</code>.</p> <pre><code># curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5-lts.sh | sh </code></pre> <p>You can find more information about the installation script in <a href="https://docs.fluentd.org/installation">Fluend Doc - Installation</a>.</p> <h3>4. Confirm if fluent-package v5 is properly installed.</h3> <pre><code>$ LANG=C yum info fluent-package Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: ftp.riken.jp * extras: ftp.riken.jp * updates: ftp.riken.jp Installed Packages Name : fluent-package Arch : x86_64 Version : 5.0.0 Release : 1.el7 Size : 64 M Repo : installed From repo : /fluent-package-5.0.0-1.el7.x86_64 Summary : The stable distribution of Fluentd URL : https://www.treasuredata.com/ License : ASL 2.0 Description : The stable distribution of Fluentd, called td-agent. </code></pre> <h3>5. Reload fluent-package daemon.</h3> <pre><code>$ sudo systemctl daemon-reload $ sudo systemctl enable --now fluentd </code></pre> <h3>6. Check installed plugins.</h3> <pre><code>$ fluent-gem list |grep fluent-plugin* fluent-plugin-calyptia-monitoring (0.1.3) fluent-plugin-elasticsearch (5.3.0) fluent-plugin-flowcounter-simple (0.1.0) fluent-plugin-kafka (0.19.0) fluent-plugin-metrics-cmetrics (0.1.2) fluent-plugin-opensearch (1.1.0) fluent-plugin-prometheus (2.0.3) fluent-plugin-prometheus_pushgateway (0.1.0) fluent-plugin-record-modifier (2.1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.2) fluent-plugin-sd-dns (0.1.0) fluent-plugin-systemd (1.0.5) fluent-plugin-td (1.2.0) fluent-plugin-utmpx (0.5.0) fluent-plugin-webhdfs (1.5.0) </code></pre> <p>You can see bundled plugins are upgraded as well but can not find plugins added on your own. In this post, added plugin was &quot;fluent-plugin-concat&quot; and it is not shown in installed list.</p> <h3>7. Install plugins added on my own.</h3> <pre><code>$ sudo fluent-gem install fluent-plugin-concat </code></pre> <pre><code>$ fluent-gem list | grep fluent-plugin* fluent-plugin-calyptia-monitoring (0.1.3) fluent-plugin-concat (2.5.0) fluent-plugin-elasticsearch (5.3.0) fluent-plugin-flowcounter-simple (0.1.0) fluent-plugin-kafka (0.19.0) fluent-plugin-metrics-cmetrics (0.1.2) fluent-plugin-opensearch (1.1.0) fluent-plugin-prometheus (2.0.3) fluent-plugin-prometheus_pushgateway (0.1.0) fluent-plugin-record-modifier (2.1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.2) fluent-plugin-sd-dns (0.1.0) fluent-plugin-systemd (1.0.5) fluent-plugin-td (1.2.0) fluent-plugin-utmpx (0.5.0) fluent-plugin-webhdfs (1.5.0) </code></pre> <p>As for fluent-package v5, &quot;fluent-plugin-concat&quot; was installed under &quot;/opt/fluent/lib/ruby/gems/3.2.0/gems/&quot; directories.</p> <pre><code>$ ls -l /opt/fluent/lib/ruby/gems/3.2.0/gems/ |grep fluent-plugin* drwxr-xr-x. 5 root root 175 7月 14 03:14 fluent-plugin-calyptia-monitoring-0.1.3 drwxr-xr-x. 4 root root 206 7月 14 03:16 fluent-plugin-concat-2.5.0 drwxr-xr-x. 5 root root 4096 7月 14 03:14 fluent-plugin-elasticsearch-5.3.0 drwxr-xr-x. 4 root root 205 7月 14 03:14 fluent-plugin-flowcounter-simple-0.1.0 drwxr-xr-x. 6 root root 191 7月 14 03:14 fluent-plugin-kafka-0.19.0 drwxr-xr-x. 5 root root 190 7月 14 03:14 fluent-plugin-metrics-cmetrics-0.1.2 drwxr-xr-x. 5 root root 4096 7月 14 03:14 fluent-plugin-opensearch-1.1.0 drwxr-xr-x. 5 root root 215 7月 14 03:14 fluent-plugin-prometheus-2.0.3 drwxr-xr-x. 6 root root 238 7月 14 03:14 fluent-plugin-prometheus_pushgateway-0.1.0 drwxr-xr-x. 4 root root 176 7月 14 03:14 fluent-plugin-record-modifier-2.1.1 drwxr-xr-x. 3 root root 210 7月 14 03:14 fluent-plugin-rewrite-tag-filter-2.4.0 drwxr-xr-x. 5 root root 230 7月 14 03:14 fluent-plugin-s3-1.7.2 drwxr-xr-x. 3 root root 170 7月 14 03:14 fluent-plugin-sd-dns-0.1.0 drwxr-xr-x. 3 root root 49 7月 14 03:14 fluent-plugin-systemd-1.0.5 drwxr-xr-x. 5 root root 221 7月 14 03:14 fluent-plugin-td-1.2.0 drwxr-xr-x. 5 root root 166 7月 14 03:14 fluent-plugin-utmpx-0.5.0 drwxr-xr-x. 4 root root 191 7月 14 03:14 fluent-plugin-webhdfs-1.5.0 </code></pre> <h3>8. Start fluent-package v5 daemon.</h3> <pre><code>$ sudo systemctl start fluentd </code></pre> <h3>9. Check if there are no error messages in fluentd logs.</h3> <pre><code>$ tail -100f /var/log/fluent/fluentd.log </code></pre> <p>Now, upgrading steps are completed. Happy Logging!</p> <h2>Additional hints for v4 users</h2> <h3>For Debian/Ubuntu</h3> <ul> <li><code>fluentd-apt-source</code> package will be marked as a transitional package. you can remove it safely with <code>sudo apt purge fluentd-apt-source</code>.</li> <li>If you want to enable <code>td-agent.service</code>, you must explicitly execute the following commands:</li> </ul> <pre><code> $ sudo systemctl unmask td-agent $ sudo systemctl enable fluentd </code></pre> <h3>For RHEL</h3> <ul> <li>If you want to enable <code>td-agent.service</code>, you must explicitly execute the following command:</li> </ul> <pre><code> $ sudo systemctl enable fluentd </code></pre> <h3>For Windows</h3> <ul> <li><code>fluent-package</code> installer was changed not to start service by default. If you want to start <code>fluentd</code> as a service, execute the following command with administrator privileges.</li> </ul> <pre><code> c:\opt\fluent&gt; net start fluentdwinsvc </code></pre> <h3>For macOS</h3> <p>WARNING: Currently we have no plan to officially support dmg version of <code>fluent-package</code> yet. It is just modified to be a minimally buildable state, it is for testing purpose only.</p> 2023-07-31 <p><code>fluent-package</code> &quot;v5&quot; is available since August 2023. <code>fluent-package</code> is the successor of <code>td-agent</code> &quot;v4&quot;.</p> <p>In this post, we will share the steps we&#39;ve tested and hopefully this will help your experience from v4 to v5.</p> <h2>Why package was renamed?</h2> <p><code>fluent-package</code> was formerly known as <code>td-agent</code>. In the past, <a href="http://www.treasuredata.com/">Treasure Data, Inc</a> took the initiative to provide the package, but now the Fluentd community does it. This is why the package name was changed.</p> <p>To represent &quot;All in one package of Fluentd which contains Fluentd and related gem packages&quot;, the package name was changed to <code>fluent-package</code>.</p> <p>Even though package name was changed, Treasure Data, Inc still sponsor the package delivery resources.</p> <h2>Which channel should I use?</h2> <p>There are two channels for <code>fluent-package</code>.</p> <ul> <li>Normal release version</li> <li>Long Term Support version</li> </ul> <p>One is normal release version which will be updated regularly - rapid release development style (<code>td-agent v4</code> was released like this in almost every quarter). In this version, fluentd will be eventually updated to newer minor version (e.g. 1.17.x and so on)</p> <p>The other is a more conservative maintenance version (Long Term Support) which will not introduce a new feature. It only applies teeny update such as security or bug fix only. LTS for v5 will be supported until March, 2025.</p> <p>More details about difference between normal release version and LTS version will be explained in <a href="fluent-package-scheduled-lifecycle">Scheduled support lifecycle announcement about Fluent Package</a>.</p> <h2>Differences between td-agent v4 and fluent-package v5</h2> <p>In the <code>fluent-package</code> v5, core components like ruby (2.7.8 -&gt; 3.2.2) and OpenSSL (1.1.1 -&gt; 3.1.0 for Windows, 3.0.8 for macOS) were updated.</p> <p>The major changes are as follows.</p> <ul> <li><code>td-agent</code> command is renamed to <code>fluentd</code>. <ul> <li><code>$ td-agent --version</code> -&gt; <code>$ fluentd --version</code></li> </ul></li> <li><code>td-agent-gem</code> command is renamed to <code>fluent-gem</code>. <ul> <li><code>$ td-agent-gem list</code> -&gt; <code>$ fluent-gem list</code></li> </ul></li> <li>The service name for non-Windows <code>td-agent</code> is renamed to <code>fluentd</code>. <ul> <li><code>$ systemctl status td-agent</code> -&gt; <code>$ systemctl status fluentd</code></li> </ul></li> </ul> <p>With the change of package name, install path, service name (e.g. /opt/fluent, fluentd.service) and so on were also changed. Basically, for <code>td-agent</code> v4 users, it aims to keep compatibility as far as possible by executing the migration process with copying old files or providing symbolic links for it.</p> <p>If you want to know the details of upgraded components, see <a href="https://github.com/fluent/fluent-package-builder/blob/master/CHANGELOG.md#release-v500---20230729">CHANGELOG.md</a>.</p> <p>NOTE: We explain for platform specific issue as &quot;Additional hints for v4 users&quot; section below.</p> <h2>Upgrade steps</h2> <p>During the upgrade process, plugins bundled in <code>td-agent</code> are automatically upgraded. With that being said, other plugins added on your own are not included. You should review if you need to upgrade plugins since some directory structures from v4 and v5 are changed.</p> <p>In this post, I will show steps with plugins added on my own, <strong>&quot;fluent-plugin-concat&quot;</strong> for instance. Here is sample configuration file I used through steps.</p> <pre><code>&lt;filter docker.log&gt; @type concat key message multiline_start_regexp /^Start/ &lt;/filter&gt; </code></pre> <h3>1. Review what plugins are installed together with td-agent v4.</h3> <pre><code>$ td-agent-gem list | grep fluent-plugin* fluent-plugin-calyptia-monitoring (0.1.3) fluent-plugin-concat (2.5.0) fluent-plugin-elasticsearch (5.3.0) fluent-plugin-flowcounter-simple (0.1.0) fluent-plugin-kafka (0.19.0) fluent-plugin-metrics-cmetrics (0.1.2) fluent-plugin-opensearch (1.1.0) fluent-plugin-prometheus (2.0.3) fluent-plugin-prometheus_pushgateway (0.1.0) fluent-plugin-record-modifier (2.1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.2) fluent-plugin-sd-dns (0.1.0) fluent-plugin-systemd (1.0.5) fluent-plugin-td (1.2.0) fluent-plugin-utmpx (0.5.0) fluent-plugin-webhdfs (1.5.0) </code></pre> <p>You can also find installed plugins under <code>/opt/td-agent/lib/ruby/gems/2.7.0/gems/</code> directories.</p> <pre><code>$ ls -l /opt/td-agent/lib/ruby/gems/2.7.0/gems |grep fluent-plugin* drwxr-xr-x. 5 root root 175 7月 14 03:01 fluent-plugin-calyptia-monitoring-0.1.3 drwxr-xr-x. 4 root root 206 7月 14 03:03 fluent-plugin-concat-2.5.0 drwxr-xr-x. 5 root root 4096 7月 14 03:01 fluent-plugin-elasticsearch-5.3.0 drwxr-xr-x. 4 root root 205 7月 14 03:01 fluent-plugin-flowcounter-simple-0.1.0 drwxr-xr-x. 6 root root 191 7月 14 03:01 fluent-plugin-kafka-0.19.0 drwxr-xr-x. 5 root root 190 7月 14 03:01 fluent-plugin-metrics-cmetrics-0.1.2 drwxr-xr-x. 5 root root 4096 7月 14 03:01 fluent-plugin-opensearch-1.1.0 drwxr-xr-x. 5 root root 215 7月 14 03:01 fluent-plugin-prometheus-2.0.3 drwxr-xr-x. 6 root root 238 7月 14 03:01 fluent-plugin-prometheus_pushgateway-0.1.0 drwxr-xr-x. 4 root root 176 7月 14 03:01 fluent-plugin-record-modifier-2.1.1 drwxr-xr-x. 3 root root 210 7月 14 03:01 fluent-plugin-rewrite-tag-filter-2.4.0 drwxr-xr-x. 5 root root 230 7月 14 03:01 fluent-plugin-s3-1.7.2 drwxr-xr-x. 3 root root 170 7月 14 03:01 fluent-plugin-sd-dns-0.1.0 drwxr-xr-x. 3 root root 49 7月 14 03:01 fluent-plugin-systemd-1.0.5 drwxr-xr-x. 5 root root 221 7月 14 03:01 fluent-plugin-td-1.2.0 drwxr-xr-x. 5 root root 166 7月 14 03:01 fluent-plugin-utmpx-0.5.0 drwxr-xr-x. 4 root root 191 7月 14 03:01 fluent-plugin-webhdfs-1.5.0 </code></pre> <h3>2. Stop td-agent v4 daemon.</h3> <pre><code>$ sudo systemctl stop td-agent </code></pre> <p>Even though <code>fluent-package</code> supports upgrade without stopping service, but recommend to stop explicitly.</p> <h3>3. Run installation script of fluent-package v5.</h3> <p>When you use RedHat or derivative distributions, you can run following script if you want to install normal release version of <code>fluent-package</code>.</p> <pre><code># curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5.sh | sh </code></pre> <p>When you use RedHat or derivative distributions, you can run following script if you want to install LTS (Long term support) version of <code>fluent-package</code>.</p> <pre><code># curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5-lts.sh | sh </code></pre> <p>You can find more information about the installation script in <a href="https://docs.fluentd.org/installation">Fluend Doc - Installation</a>.</p> <h3>4. Confirm if fluent-package v5 is properly installed.</h3> <pre><code>$ LANG=C yum info fluent-package Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: ftp.riken.jp * extras: ftp.riken.jp * updates: ftp.riken.jp Installed Packages Name : fluent-package Arch : x86_64 Version : 5.0.0 Release : 1.el7 Size : 64 M Repo : installed From repo : /fluent-package-5.0.0-1.el7.x86_64 Summary : The stable distribution of Fluentd URL : https://www.treasuredata.com/ License : ASL 2.0 Description : The stable distribution of Fluentd, called td-agent. </code></pre> <h3>5. Reload fluent-package daemon.</h3> <pre><code>$ sudo systemctl daemon-reload $ sudo systemctl enable --now fluentd </code></pre> <h3>6. Check installed plugins.</h3> <pre><code>$ fluent-gem list |grep fluent-plugin* fluent-plugin-calyptia-monitoring (0.1.3) fluent-plugin-elasticsearch (5.3.0) fluent-plugin-flowcounter-simple (0.1.0) fluent-plugin-kafka (0.19.0) fluent-plugin-metrics-cmetrics (0.1.2) fluent-plugin-opensearch (1.1.0) fluent-plugin-prometheus (2.0.3) fluent-plugin-prometheus_pushgateway (0.1.0) fluent-plugin-record-modifier (2.1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.2) fluent-plugin-sd-dns (0.1.0) fluent-plugin-systemd (1.0.5) fluent-plugin-td (1.2.0) fluent-plugin-utmpx (0.5.0) fluent-plugin-webhdfs (1.5.0) </code></pre> <p>You can see bundled plugins are upgraded as well but can not find plugins added on your own. In this post, added plugin was &quot;fluent-plugin-concat&quot; and it is not shown in installed list.</p> <h3>7. Install plugins added on my own.</h3> <pre><code>$ sudo fluent-gem install fluent-plugin-concat </code></pre> <pre><code>$ fluent-gem list | grep fluent-plugin* fluent-plugin-calyptia-monitoring (0.1.3) fluent-plugin-concat (2.5.0) fluent-plugin-elasticsearch (5.3.0) fluent-plugin-flowcounter-simple (0.1.0) fluent-plugin-kafka (0.19.0) fluent-plugin-metrics-cmetrics (0.1.2) fluent-plugin-opensearch (1.1.0) fluent-plugin-prometheus (2.0.3) fluent-plugin-prometheus_pushgateway (0.1.0) fluent-plugin-record-modifier (2.1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.7.2) fluent-plugin-sd-dns (0.1.0) fluent-plugin-systemd (1.0.5) fluent-plugin-td (1.2.0) fluent-plugin-utmpx (0.5.0) fluent-plugin-webhdfs (1.5.0) </code></pre> <p>As for fluent-package v5, &quot;fluent-plugin-concat&quot; was installed under &quot;/opt/fluent/lib/ruby/gems/3.2.0/gems/&quot; directories.</p> <pre><code>$ ls -l /opt/fluent/lib/ruby/gems/3.2.0/gems/ |grep fluent-plugin* drwxr-xr-x. 5 root root 175 7月 14 03:14 fluent-plugin-calyptia-monitoring-0.1.3 drwxr-xr-x. 4 root root 206 7月 14 03:16 fluent-plugin-concat-2.5.0 drwxr-xr-x. 5 root root 4096 7月 14 03:14 fluent-plugin-elasticsearch-5.3.0 drwxr-xr-x. 4 root root 205 7月 14 03:14 fluent-plugin-flowcounter-simple-0.1.0 drwxr-xr-x. 6 root root 191 7月 14 03:14 fluent-plugin-kafka-0.19.0 drwxr-xr-x. 5 root root 190 7月 14 03:14 fluent-plugin-metrics-cmetrics-0.1.2 drwxr-xr-x. 5 root root 4096 7月 14 03:14 fluent-plugin-opensearch-1.1.0 drwxr-xr-x. 5 root root 215 7月 14 03:14 fluent-plugin-prometheus-2.0.3 drwxr-xr-x. 6 root root 238 7月 14 03:14 fluent-plugin-prometheus_pushgateway-0.1.0 drwxr-xr-x. 4 root root 176 7月 14 03:14 fluent-plugin-record-modifier-2.1.1 drwxr-xr-x. 3 root root 210 7月 14 03:14 fluent-plugin-rewrite-tag-filter-2.4.0 drwxr-xr-x. 5 root root 230 7月 14 03:14 fluent-plugin-s3-1.7.2 drwxr-xr-x. 3 root root 170 7月 14 03:14 fluent-plugin-sd-dns-0.1.0 drwxr-xr-x. 3 root root 49 7月 14 03:14 fluent-plugin-systemd-1.0.5 drwxr-xr-x. 5 root root 221 7月 14 03:14 fluent-plugin-td-1.2.0 drwxr-xr-x. 5 root root 166 7月 14 03:14 fluent-plugin-utmpx-0.5.0 drwxr-xr-x. 4 root root 191 7月 14 03:14 fluent-plugin-webhdfs-1.5.0 </code></pre> <h3>8. Start fluent-package v5 daemon.</h3> <pre><code>$ sudo systemctl start fluentd </code></pre> <h3>9. Check if there are no error messages in fluentd logs.</h3> <pre><code>$ tail -100f /var/log/fluent/fluentd.log </code></pre> <p>Now, upgrading steps are completed. Happy Logging!</p> <h2>Additional hints for v4 users</h2> <h3>For Debian/Ubuntu</h3> <ul> <li><code>fluentd-apt-source</code> package will be marked as a transitional package. you can remove it safely with <code>sudo apt purge fluentd-apt-source</code>.</li> <li>If you want to enable <code>td-agent.service</code>, you must explicitly execute the following commands:</li> </ul> <pre><code> $ sudo systemctl unmask td-agent $ sudo systemctl enable fluentd </code></pre> <h3>For RHEL</h3> <ul> <li>If you want to enable <code>td-agent.service</code>, you must explicitly execute the following command:</li> </ul> <pre><code> $ sudo systemctl enable fluentd </code></pre> <h3>For Windows</h3> <ul> <li><code>fluent-package</code> installer was changed not to start service by default. If you want to start <code>fluentd</code> as a service, execute the following command with administrator privileges.</li> </ul> <pre><code> c:\opt\fluent&gt; net start fluentdwinsvc </code></pre> <h3>For macOS</h3> <p>WARNING: Currently we have no plan to officially support dmg version of <code>fluent-package</code> yet. It is just modified to be a minimally buildable state, it is for testing purpose only.</p> td-agent v4.5.0 has been released https://www.fluentd.org/blog/td-agent-v4.5.0-has-been-released <p>Hi users!</p> <p>We have released td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.5.0">v4.5.0</a>. td-agent is a stable distribution package of Fluentd.</p> <h3>Changes from td-agent v4.4.2</h3> <ul> <li>Update bundled Ruby to 2.7.8 which is the last version of Ruby 2.7 series. <ul> <li>Only for RHEL 9 &amp; Ubuntu 22.04 jammy, we updat Ruby to 3.1.4 exceptionally.</li> </ul></li> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.1">1.16.1</a></li> <li>Update plugins <ul> <li>fluent-plugin-kafka 0.19.0</li> <li>fluent-plugin-elasticsearch 5.3.0</li> <li>fluent-plugin-opensearch 1.1.0</li> </ul></li> <li>Drop Debian buster support</li> </ul> <h3>About next major version</h3> <p>We are preparing next major version of td-agent which bundles Ruby 3.2. It&#39;s still work in progerss, we&#39;ll announce it as soon as we are ready for it.</p> <p>Please note that the package name will be changed to <code>fluent-package</code>. Please see the following discussion for more detail: <a href="https://github.com/fluent/fluentd/discussions/3860">https://github.com/fluent/fluentd/discussions/3860</a></p> <h3>Download</h3> <p>Please see <a href="/download#td-agent">the download page</a>.</p> 2023-05-08 <p>Hi users!</p> <p>We have released td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.5.0">v4.5.0</a>. td-agent is a stable distribution package of Fluentd.</p> <h3>Changes from td-agent v4.4.2</h3> <ul> <li>Update bundled Ruby to 2.7.8 which is the last version of Ruby 2.7 series. <ul> <li>Only for RHEL 9 &amp; Ubuntu 22.04 jammy, we updat Ruby to 3.1.4 exceptionally.</li> </ul></li> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.1">1.16.1</a></li> <li>Update plugins <ul> <li>fluent-plugin-kafka 0.19.0</li> <li>fluent-plugin-elasticsearch 5.3.0</li> <li>fluent-plugin-opensearch 1.1.0</li> </ul></li> <li>Drop Debian buster support</li> </ul> <h3>About next major version</h3> <p>We are preparing next major version of td-agent which bundles Ruby 3.2. It&#39;s still work in progerss, we&#39;ll announce it as soon as we are ready for it.</p> <p>Please note that the package name will be changed to <code>fluent-package</code>. Please see the following discussion for more detail: <a href="https://github.com/fluent/fluentd/discussions/3860">https://github.com/fluent/fluentd/discussions/3860</a></p> <h3>Download</h3> <p>Please see <a href="/download#td-agent">the download page</a>.</p> Fluentd v1.16.1 has been released https://www.fluentd.org/blog/fluentd-v1.16.1-has-been-released <p>Hi users!</p> <p>We have released v1.16.1 on 2023-04-17. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1161---20230417">here</a>.</p> <p>In the previous version v1.16.0, there was a bug that caused Fluentd to fail to start with certain secondary configurations. In this release, we fixed that problem and added a new feature of <code>in_tcp</code>. And other some improvements!</p> <h2>Enhancement</h2> <ul> <li><code>in_tcp</code>: Add <code>message_length_limit</code> to drop large incoming data. <a href="https://github.com/fluent/fluentd/pull/4137">#4137</a></li> </ul> <h2>Bug Fixes</h2> <h3><code>out_secondary_file</code>: Fix NameError of <code>SecondaryFileOutput</code> when setting other secondaries</h3> <p>In the previous version v1.16.0, there was a bug that if you set another secondary without ever setting <code>out_secondary_file</code>, Fluentd fails to start with the following error.</p> <pre><code>/path/to/fluentd/lib/fluent/plugin/output.rb:429:in `configure&#39;: uninitialized constant Fluent::Plugin::Output::SecondaryFileOutput (NameError) if (@secondary.class != SecondaryFileOutput) &amp;&amp; ^^^^^^^^^^^^^^^^^^^ </code></pre> <p>In this release, we fixed this problem.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4124">#4124</a> for more information.</p> <h3><code>Buffer</code>: Fix that <code>compress</code> setting causes unexpected error when receiving already compressed MessagePack</h3> <p>With this release, it is now possible that Fluentd forwards already compressed MessagePack data to another Fluentd as is (without decompressing).</p> <p>A possible use case is two-stage forwarding.</p> <p>Forwarder1(<code>out_forward</code>) -&gt; Forwarder2(<code>in_forward</code>, <code>out_forward</code>) -&gt; Aggregator(<code>in_forward</code>)</p> <p>In this case, Forwarder2 should process the data of <code>CompressedMessagePackEventStream</code> as is (without decompressing) and re-forward the data to Aggregator.</p> <p>Until this version, a bug prevented us from setting Forwarder2&#39;s <code>compress gzip</code> of <a href="https://docs.fluentd.org/output/forward#how-to-enable-gzip-compression">out_forward</a> or <a href="https://docs.fluentd.org/configuration/buffer-section#buffering-parameters">buffer</a>. It resulted in the following error.</p> <pre><code>[error]: #0 unexpected error on reading data host=&quot;...&quot; port=... error_class=ArgumentError error=&quot;unknown keyword: :packer&quot; </code></pre> <p>So we had no choice but to decompress and forward it from Forwarder2 to Aggregator.</p> <p>With this release, we can now set <code>compress gzip</code> for Forwarder2!</p> <p>See <a href="https://github.com/fluent/fluentd/issues/4146">#4146</a> and <a href="https://github.com/fluent/fluentd/pull/4147">#4147</a> for more information.</p> <h3>Others</h3> <ul> <li>Server helper: Suppress error of <code>UDPServer</code> over <code>max_bytes</code> on Windows. <a href="https://github.com/fluent/fluentd/pull/4131">#4131</a></li> </ul> <p>Enjoy logging!</p> 2023-04-20 <p>Hi users!</p> <p>We have released v1.16.1 on 2023-04-17. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1161---20230417">here</a>.</p> <p>In the previous version v1.16.0, there was a bug that caused Fluentd to fail to start with certain secondary configurations. In this release, we fixed that problem and added a new feature of <code>in_tcp</code>. And other some improvements!</p> <h2>Enhancement</h2> <ul> <li><code>in_tcp</code>: Add <code>message_length_limit</code> to drop large incoming data. <a href="https://github.com/fluent/fluentd/pull/4137">#4137</a></li> </ul> <h2>Bug Fixes</h2> <h3><code>out_secondary_file</code>: Fix NameError of <code>SecondaryFileOutput</code> when setting other secondaries</h3> <p>In the previous version v1.16.0, there was a bug that if you set another secondary without ever setting <code>out_secondary_file</code>, Fluentd fails to start with the following error.</p> <pre><code>/path/to/fluentd/lib/fluent/plugin/output.rb:429:in `configure&#39;: uninitialized constant Fluent::Plugin::Output::SecondaryFileOutput (NameError) if (@secondary.class != SecondaryFileOutput) &amp;&amp; ^^^^^^^^^^^^^^^^^^^ </code></pre> <p>In this release, we fixed this problem.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4124">#4124</a> for more information.</p> <h3><code>Buffer</code>: Fix that <code>compress</code> setting causes unexpected error when receiving already compressed MessagePack</h3> <p>With this release, it is now possible that Fluentd forwards already compressed MessagePack data to another Fluentd as is (without decompressing).</p> <p>A possible use case is two-stage forwarding.</p> <p>Forwarder1(<code>out_forward</code>) -&gt; Forwarder2(<code>in_forward</code>, <code>out_forward</code>) -&gt; Aggregator(<code>in_forward</code>)</p> <p>In this case, Forwarder2 should process the data of <code>CompressedMessagePackEventStream</code> as is (without decompressing) and re-forward the data to Aggregator.</p> <p>Until this version, a bug prevented us from setting Forwarder2&#39;s <code>compress gzip</code> of <a href="https://docs.fluentd.org/output/forward#how-to-enable-gzip-compression">out_forward</a> or <a href="https://docs.fluentd.org/configuration/buffer-section#buffering-parameters">buffer</a>. It resulted in the following error.</p> <pre><code>[error]: #0 unexpected error on reading data host=&quot;...&quot; port=... error_class=ArgumentError error=&quot;unknown keyword: :packer&quot; </code></pre> <p>So we had no choice but to decompress and forward it from Forwarder2 to Aggregator.</p> <p>With this release, we can now set <code>compress gzip</code> for Forwarder2!</p> <p>See <a href="https://github.com/fluent/fluentd/issues/4146">#4146</a> and <a href="https://github.com/fluent/fluentd/pull/4147">#4147</a> for more information.</p> <h3>Others</h3> <ul> <li>Server helper: Suppress error of <code>UDPServer</code> over <code>max_bytes</code> on Windows. <a href="https://github.com/fluent/fluentd/pull/4131">#4131</a></li> </ul> <p>Enjoy logging!</p> Fluentd v1.16.0 has been released https://www.fluentd.org/blog/fluentd-v1.16.0-has-been-released <p>Hi users!</p> <p>We have released v1.16.0 on 2023-03-29. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1160---20230329">here</a>.</p> <p>This release is a new release of v1.16 series. In this release, we enhanced the feature for chunk file corruption and fixed some bugs, mainly about logging and race condition errors.</p> <h2>Enhancement</h2> <h3>Backup corrupted chunk files at resuming</h3> <p>Some chunk files in <a href="https://docs.fluentd.org/buffer/file">buffer_file</a> or <a href="https://docs.fluentd.org/buffer/file_single">buffer_file_single</a> are possibly corrupted after Fluentd stopped abnormally, such as due to a power failure.</p> <p>Fluentd has a feature to backup unrecoverable chunks as before. However, when Fluentd resumes, remaining corrupted chunk files were just deleted.</p> <p>Since this release, Fluentd also backups those files at resuming. In addition, Fluentd outputs some useful logs about possibly corrupted chunk files due to an abnormal termination.</p> <p>Please see <a href="https://docs.fluentd.org/buffer#detecting-chunk-file-corruption-when-fluentd-starts-up">the document</a> and the following PRs for more information.</p> <ul> <li><a href="https://github.com/fluent/fluentd/pull/4025">buffer: backup broken file chunk</a></li> <li><a href="https://github.com/fluent/fluentd/pull/4027">Add warning messages for restoring buffer with <code>flush_at_shutdown true</code></a></li> <li><a href="https://github.com/fluent/fluentd/pull/4028">Add logs for time period of restored buffer possibly broken</a></li> </ul> <h3><code>in_tcp</code>: Add <code>send_keepalive_packet</code> option</h3> <p>Since this release, you can use <a href="https://docs.fluentd.org/input/tcp#general-configuration">send_keepalive_packet option</a> for <code>in_tcp</code>, as well as <a href="https://docs.fluentd.org/input/forward">in_forward</a> or <a href="https://docs.fluentd.org/input/syslog">in_syslog</a>.</p> <p>Please see <a href="https://docs.fluentd.org/input/tcp#general-configuration">the document</a> and <a href="https://github.com/fluent/fluentd/pull/3961">#3961</a> for more information.</p> <h2>Bug Fixes</h2> <ul> <li><code>out_secondary_file</code>: Fix race condition. <a href="https://github.com/fluent/fluentd/pull/4081">#4081</a></li> <li><code>out_secondary_file</code>: Suppress warning using different secondary for out<em>secondary</em>file. <a href="https://github.com/fluent/fluentd/pull/4087">#4087</a></li> <li><code>out_forward</code>: Fix error of ack handling conflict with <code>require_ack_response</code>. <a href="https://github.com/fluent/fluentd/pull/4030">#4030</a></li> <li>Windows: Fix bug that Fluentd sometimes tries to use an unavailable port and fails to start. <a href="https://github.com/fluent/fluentd/pull/4092">#4092</a></li> <li>Windows: Fix bug that flushing or graceful reloading cause reopening wrong log file. <a href="https://github.com/fluent/fluentd/pull/4054">#4054</a></li> <li>Fix not to log some initial messages without <code>format</code> setting applied. <a href="https://github.com/fluent/fluentd/pull/4091">#4091</a></li> <li>Fix bug that some <code>system</code> configs are not reflected. <a href="https://github.com/fluent/fluentd/pull/4064">#4064</a>, <a href="https://github.com/fluent/fluentd/pull/4065">#4065</a> <ul> <li><a href="https://docs.fluentd.org/deployment/system-config#suppress_repeated_stacktrace">suppress_repeated_stacktrace</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#ignore_same_log_interval">ignore_same_log_interval</a> (It was not reflected only to the supervisor process)</li> </ul></li> <li>Non-Windows: Fix not to generate sigdump file after receiving <code>SIGTERM</code>. <a href="https://github.com/fluent/fluentd/pull/4043">#4043</a></li> <li>Fix bug that <code>system_config.workers</code> value can be wrongly <code>1</code> at config check at startup. <a href="https://github.com/fluent/fluentd/pull/4066">#4066</a></li> <li>Fix bug that a variable whose type is not <code>Fluent::Config::Element</code> can be passed to <code>Fluent::Plugin::Base::configure()</code>. <a href="https://github.com/fluent/fluentd/pull/4066">#4066</a></li> </ul> <p>Enjoy logging!</p> 2023-03-29 <p>Hi users!</p> <p>We have released v1.16.0 on 2023-03-29. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1160---20230329">here</a>.</p> <p>This release is a new release of v1.16 series. In this release, we enhanced the feature for chunk file corruption and fixed some bugs, mainly about logging and race condition errors.</p> <h2>Enhancement</h2> <h3>Backup corrupted chunk files at resuming</h3> <p>Some chunk files in <a href="https://docs.fluentd.org/buffer/file">buffer_file</a> or <a href="https://docs.fluentd.org/buffer/file_single">buffer_file_single</a> are possibly corrupted after Fluentd stopped abnormally, such as due to a power failure.</p> <p>Fluentd has a feature to backup unrecoverable chunks as before. However, when Fluentd resumes, remaining corrupted chunk files were just deleted.</p> <p>Since this release, Fluentd also backups those files at resuming. In addition, Fluentd outputs some useful logs about possibly corrupted chunk files due to an abnormal termination.</p> <p>Please see <a href="https://docs.fluentd.org/buffer#detecting-chunk-file-corruption-when-fluentd-starts-up">the document</a> and the following PRs for more information.</p> <ul> <li><a href="https://github.com/fluent/fluentd/pull/4025">buffer: backup broken file chunk</a></li> <li><a href="https://github.com/fluent/fluentd/pull/4027">Add warning messages for restoring buffer with <code>flush_at_shutdown true</code></a></li> <li><a href="https://github.com/fluent/fluentd/pull/4028">Add logs for time period of restored buffer possibly broken</a></li> </ul> <h3><code>in_tcp</code>: Add <code>send_keepalive_packet</code> option</h3> <p>Since this release, you can use <a href="https://docs.fluentd.org/input/tcp#general-configuration">send_keepalive_packet option</a> for <code>in_tcp</code>, as well as <a href="https://docs.fluentd.org/input/forward">in_forward</a> or <a href="https://docs.fluentd.org/input/syslog">in_syslog</a>.</p> <p>Please see <a href="https://docs.fluentd.org/input/tcp#general-configuration">the document</a> and <a href="https://github.com/fluent/fluentd/pull/3961">#3961</a> for more information.</p> <h2>Bug Fixes</h2> <ul> <li><code>out_secondary_file</code>: Fix race condition. <a href="https://github.com/fluent/fluentd/pull/4081">#4081</a></li> <li><code>out_secondary_file</code>: Suppress warning using different secondary for out<em>secondary</em>file. <a href="https://github.com/fluent/fluentd/pull/4087">#4087</a></li> <li><code>out_forward</code>: Fix error of ack handling conflict with <code>require_ack_response</code>. <a href="https://github.com/fluent/fluentd/pull/4030">#4030</a></li> <li>Windows: Fix bug that Fluentd sometimes tries to use an unavailable port and fails to start. <a href="https://github.com/fluent/fluentd/pull/4092">#4092</a></li> <li>Windows: Fix bug that flushing or graceful reloading cause reopening wrong log file. <a href="https://github.com/fluent/fluentd/pull/4054">#4054</a></li> <li>Fix not to log some initial messages without <code>format</code> setting applied. <a href="https://github.com/fluent/fluentd/pull/4091">#4091</a></li> <li>Fix bug that some <code>system</code> configs are not reflected. <a href="https://github.com/fluent/fluentd/pull/4064">#4064</a>, <a href="https://github.com/fluent/fluentd/pull/4065">#4065</a> <ul> <li><a href="https://docs.fluentd.org/deployment/system-config#suppress_repeated_stacktrace">suppress_repeated_stacktrace</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#ignore_same_log_interval">ignore_same_log_interval</a> (It was not reflected only to the supervisor process)</li> </ul></li> <li>Non-Windows: Fix not to generate sigdump file after receiving <code>SIGTERM</code>. <a href="https://github.com/fluent/fluentd/pull/4043">#4043</a></li> <li>Fix bug that <code>system_config.workers</code> value can be wrongly <code>1</code> at config check at startup. <a href="https://github.com/fluent/fluentd/pull/4066">#4066</a></li> <li>Fix bug that a variable whose type is not <code>Fluent::Config::Element</code> can be passed to <code>Fluent::Plugin::Base::configure()</code>. <a href="https://github.com/fluent/fluentd/pull/4066">#4066</a></li> </ul> <p>Enjoy logging!</p> Fluentd v1.15.3 has been released https://www.fluentd.org/blog/fluentd-v1.15.3-has-been-released <p>Hi users!</p> <p>We have released v1.15.3 on 2022-11-02. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1153---20221102">here</a>.</p> <p>In this release, we fixed several bugs.</p> <h2>Bug Fixes</h2> <h3><code>out_file</code>: Fix the multi-worker check with <code>&lt;worker N-M&gt;</code> directive</h3> <p>In the v1.15.1, we added a new feature to prevent write conflicts between multiple workers.</p> <ul> <li><a href="https://github.com/fluent/fluentd/pull/3808">#3808</a></li> </ul> <p>However, this fix didn&#39;t work correctly with <code>&lt;worker N-M&gt;</code> directive.</p> <ul> <li><a href="https://docs.fluentd.org/deployment/multi-process-workers#less-than-worker-n-m-greater-than-directive"><code>&lt;worker N-M&gt;</code> directive</a></li> </ul> <p>With this setting, the configuration check before starting the supervisor mistakenly determines that the setting of <code>out_file</code> is invalid and causes Fluentd to terminate.</p> <p>In this release, we fixed this problem.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3942">#3942</a> for more information.</p> <h3>Fix log rotation on Windows</h3> <p>We can configure log rotation setting as follows.</p> <ul> <li><a href="https://docs.fluentd.org/deployment/logging#log-rotation-setting">Log Rotation Setting</a></li> </ul> <p>On Windows, the log files must be separated by each process.</p> <blockquote> <p>NOTE: When --log-rotate-age is specified on Windows, log files are separated into log-supervisor-0.log, log-0.log, ..., log-N.log where N is generation - 1 due to the system limitation. Windows does not permit delete and rename files simultaneously owned by another process.</p> </blockquote> <p>However, that log rotation setting in the system-directive didn&#39;t satisfy this and Fluentd outputted the logs of the supervisor process into the log file of the worker-0 process.</p> <p>This could cause errors during log rotation.</p> <p>In this release, we fixed this problem.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3939">#3939</a> for more information.</p> <h3>Support glob for <code>!include</code> directive in YAML config format</h3> <p>In the v1.15.0, we supported YAML config format.</p> <ul> <li><a href="https://docs.fluentd.org/configuration/config-file-yaml">Config File Syntax (YAML)</a></li> <li><a href="https://github.com/fluent/fluentd/pull/3712">#3712</a></li> </ul> <p>However, handling glob of <code>!include</code> directive was leaked.</p> <p>In this release, we supported it.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3917">#3917</a> for more information.</p> <h3>Remove meaningless and unsecure <code>oj</code> options</h3> <p>In this release, we removed the following <code>oj</code> options.</p> <ul> <li><code>FLUENT_OJ_OPTION_MODE=object</code></li> <li><code>FLUENT_OJ_OPTION_MAX_NESTING</code></li> </ul> <p><code>FLUENT_OJ_OPTION_MODE=object</code> was unsecure, so we removed it.</p> <ul> <li><a href="https://github.com/fluent/fluentd/security/advisories/GHSA-fppq-mj76-fpj2">GHSA-fppq-mj76-fpj2</a></li> </ul> <p>This option is not used unless a user explicitly sets the environmental variable, so almost all users do not need to care about this vulnerability.</p> <p><code>FLUENT_OJ_OPTION_MAX_NESTING</code> was not supported, so we removed it.</p> <ul> <li><a href="https://github.com/fluent/fluentd/issues/3311#issuecomment-923741031">https://github.com/fluent/fluentd/issues/3311#issuecomment-923741031</a></li> </ul> <p>See <a href="https://github.com/fluent/fluentd/pull/3929">#3929</a> for more information.</p> <p>Enjoy logging!</p> 2022-11-11 <p>Hi users!</p> <p>We have released v1.15.3 on 2022-11-02. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1153---20221102">here</a>.</p> <p>In this release, we fixed several bugs.</p> <h2>Bug Fixes</h2> <h3><code>out_file</code>: Fix the multi-worker check with <code>&lt;worker N-M&gt;</code> directive</h3> <p>In the v1.15.1, we added a new feature to prevent write conflicts between multiple workers.</p> <ul> <li><a href="https://github.com/fluent/fluentd/pull/3808">#3808</a></li> </ul> <p>However, this fix didn&#39;t work correctly with <code>&lt;worker N-M&gt;</code> directive.</p> <ul> <li><a href="https://docs.fluentd.org/deployment/multi-process-workers#less-than-worker-n-m-greater-than-directive"><code>&lt;worker N-M&gt;</code> directive</a></li> </ul> <p>With this setting, the configuration check before starting the supervisor mistakenly determines that the setting of <code>out_file</code> is invalid and causes Fluentd to terminate.</p> <p>In this release, we fixed this problem.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3942">#3942</a> for more information.</p> <h3>Fix log rotation on Windows</h3> <p>We can configure log rotation setting as follows.</p> <ul> <li><a href="https://docs.fluentd.org/deployment/logging#log-rotation-setting">Log Rotation Setting</a></li> </ul> <p>On Windows, the log files must be separated by each process.</p> <blockquote> <p>NOTE: When --log-rotate-age is specified on Windows, log files are separated into log-supervisor-0.log, log-0.log, ..., log-N.log where N is generation - 1 due to the system limitation. Windows does not permit delete and rename files simultaneously owned by another process.</p> </blockquote> <p>However, that log rotation setting in the system-directive didn&#39;t satisfy this and Fluentd outputted the logs of the supervisor process into the log file of the worker-0 process.</p> <p>This could cause errors during log rotation.</p> <p>In this release, we fixed this problem.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3939">#3939</a> for more information.</p> <h3>Support glob for <code>!include</code> directive in YAML config format</h3> <p>In the v1.15.0, we supported YAML config format.</p> <ul> <li><a href="https://docs.fluentd.org/configuration/config-file-yaml">Config File Syntax (YAML)</a></li> <li><a href="https://github.com/fluent/fluentd/pull/3712">#3712</a></li> </ul> <p>However, handling glob of <code>!include</code> directive was leaked.</p> <p>In this release, we supported it.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3917">#3917</a> for more information.</p> <h3>Remove meaningless and unsecure <code>oj</code> options</h3> <p>In this release, we removed the following <code>oj</code> options.</p> <ul> <li><code>FLUENT_OJ_OPTION_MODE=object</code></li> <li><code>FLUENT_OJ_OPTION_MAX_NESTING</code></li> </ul> <p><code>FLUENT_OJ_OPTION_MODE=object</code> was unsecure, so we removed it.</p> <ul> <li><a href="https://github.com/fluent/fluentd/security/advisories/GHSA-fppq-mj76-fpj2">GHSA-fppq-mj76-fpj2</a></li> </ul> <p>This option is not used unless a user explicitly sets the environmental variable, so almost all users do not need to care about this vulnerability.</p> <p><code>FLUENT_OJ_OPTION_MAX_NESTING</code> was not supported, so we removed it.</p> <ul> <li><a href="https://github.com/fluent/fluentd/issues/3311#issuecomment-923741031">https://github.com/fluent/fluentd/issues/3311#issuecomment-923741031</a></li> </ul> <p>See <a href="https://github.com/fluent/fluentd/pull/3929">#3929</a> for more information.</p> <p>Enjoy logging!</p> td-agent v4.4.2 has been released https://www.fluentd.org/blog/td-agent-v4.4.2-has-been-released <p>Hi users!</p> <p>We have released td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.4.2">v4.4.2</a>. td-agent is a stable distribution package of Fluentd.</p> <h3>Changes from td-agent v4.4.1</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.15.3">1.15.3</a> <ul> <li>Fix log initializer to correctly create per-process files on Windows</li> <li>out_file: Fix the multi-worker check with <code>&lt;worker 0-N&gt;</code> directive</li> <li>Remove meaningless oj options</li> <li>Support glob for <code>!include</code> directive in YAML config format</li> </ul></li> <li>macOS: Support Apple Silicon</li> <li>Update plugins <ul> <li>fluent-plugin-elasticsearch 5.2.4</li> <li>fluent-plugin-opensearch 1.0.8</li> <li>fluent-plugin-s3 1.7.2</li> <li>fluent-plugin-windows-eventlog 0.8.2</li> </ul></li> </ul> <h3>Download</h3> <p>Please see <a href="/download#td-agent">the download page</a>.</p> 2022-11-08 <p>Hi users!</p> <p>We have released td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.4.2">v4.4.2</a>. td-agent is a stable distribution package of Fluentd.</p> <h3>Changes from td-agent v4.4.1</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.15.3">1.15.3</a> <ul> <li>Fix log initializer to correctly create per-process files on Windows</li> <li>out_file: Fix the multi-worker check with <code>&lt;worker 0-N&gt;</code> directive</li> <li>Remove meaningless oj options</li> <li>Support glob for <code>!include</code> directive in YAML config format</li> </ul></li> <li>macOS: Support Apple Silicon</li> <li>Update plugins <ul> <li>fluent-plugin-elasticsearch 5.2.4</li> <li>fluent-plugin-opensearch 1.0.8</li> <li>fluent-plugin-s3 1.7.2</li> <li>fluent-plugin-windows-eventlog 0.8.2</li> </ul></li> </ul> <h3>Download</h3> <p>Please see <a href="/download#td-agent">the download page</a>.</p>