Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More td-agent v4.4.1 has been released https://www.fluentd.org/blog/td-agent-v4.4.1-has-been-released <p>Hi users!</p> <p>We have released td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.4.1">v4.4.1</a>. td-agent is a stable distribution package of Fluentd.</p> <h3>Changes from td-agent v4.4.0</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.15.2">1.15.2</a> <ul> <li>out_file: Fix append mode with <code>--daemon</code> flag</li> <li>Drop win32-api gem dependency</li> <li>Please see also <a href="fluentd-v1.15.2-has-been-released">the release announcement of v1.15.2</a></li> </ul></li> <li>Update fluent-plugin-kafka to <a href="https://github.com/fluent/fluent-plugin-kafka/releases/tag/v0.18.1">0.18.1</a> <ul> <li>Fix out<em>kafka2 doesn&#39;t respect `chunk</em>limit<em>records<code>and</code>chunk</em>limit_size`</li> </ul></li> </ul> <h3>Download</h3> <p>Please see <a href="/download">the download page</a>.</p> 2022-08-23 <p>Hi users!</p> <p>We have released td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.4.1">v4.4.1</a>. td-agent is a stable distribution package of Fluentd.</p> <h3>Changes from td-agent v4.4.0</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.15.2">1.15.2</a> <ul> <li>out_file: Fix append mode with <code>--daemon</code> flag</li> <li>Drop win32-api gem dependency</li> <li>Please see also <a href="fluentd-v1.15.2-has-been-released">the release announcement of v1.15.2</a></li> </ul></li> <li>Update fluent-plugin-kafka to <a href="https://github.com/fluent/fluent-plugin-kafka/releases/tag/v0.18.1">0.18.1</a> <ul> <li>Fix out<em>kafka2 doesn&#39;t respect `chunk</em>limit<em>records<code>and</code>chunk</em>limit_size`</li> </ul></li> </ul> <h3>Download</h3> <p>Please see <a href="/download">the download page</a>.</p> Fluentd v1.15.2 has been released https://www.fluentd.org/blog/fluentd-v1.15.2-has-been-released <p>Hi users!</p> <p>We have released v1.15.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1152---20220822">here</a>.</p> <p>In this release, we fixed the problem that <code>out_file</code>&#39;s append mode with <code>--daemon</code> flag failed to flush the buffer.</p> <h2>Bug Fixes</h2> <h3><code>out_file</code>: Fix append mode with <code>--daemon</code> flag</h3> <p>In the previous version v1.15.1, we added a new feature to prevent write conflicts between multiple workers.</p> <ul> <li><a href="https://github.com/fluent/fluentd/pull/3808">#3808</a></li> </ul> <p>However, this fix didn&#39;t work correctly with <code>--daemon</code> flag of Fluentd command.</p> <ul> <li><a href="https://docs.fluentd.org/deployment/command-line-option">Command Line Option</a></li> </ul> <p>With that flag, <code>out_file</code> in the append mode fails to flush the buffer. This is a critical bug for td-agent, because td-agent uses <code>--daemon</code> flag by default.</p> <p>In this release, we fixed this problem.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3864">#3864</a> for more information.</p> <h2>Miscellaneous fixes</h2> <ul> <li>Add a new system configuration <code>enable_jit</code>. <a href="https://github.com/fluent/fluentd/pull/3857">#3857</a></li> <li>child_process: Plug file descriptor leak. <a href="https://github.com/fluent/fluentd/pull/3844">#3844</a></li> <li>Drop win32-api gem to support Ruby 3.2. <a href="https://github.com/fluent/fluentd/pull/3849">#3849</a>, <a href="https://github.com/fluent/fluentd/pull/3866">#3866</a></li> </ul> <p>Enjoy logging!</p> 2022-08-22 <p>Hi users!</p> <p>We have released v1.15.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1152---20220822">here</a>.</p> <p>In this release, we fixed the problem that <code>out_file</code>&#39;s append mode with <code>--daemon</code> flag failed to flush the buffer.</p> <h2>Bug Fixes</h2> <h3><code>out_file</code>: Fix append mode with <code>--daemon</code> flag</h3> <p>In the previous version v1.15.1, we added a new feature to prevent write conflicts between multiple workers.</p> <ul> <li><a href="https://github.com/fluent/fluentd/pull/3808">#3808</a></li> </ul> <p>However, this fix didn&#39;t work correctly with <code>--daemon</code> flag of Fluentd command.</p> <ul> <li><a href="https://docs.fluentd.org/deployment/command-line-option">Command Line Option</a></li> </ul> <p>With that flag, <code>out_file</code> in the append mode fails to flush the buffer. This is a critical bug for td-agent, because td-agent uses <code>--daemon</code> flag by default.</p> <p>In this release, we fixed this problem.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3864">#3864</a> for more information.</p> <h2>Miscellaneous fixes</h2> <ul> <li>Add a new system configuration <code>enable_jit</code>. <a href="https://github.com/fluent/fluentd/pull/3857">#3857</a></li> <li>child_process: Plug file descriptor leak. <a href="https://github.com/fluent/fluentd/pull/3844">#3844</a></li> <li>Drop win32-api gem to support Ruby 3.2. <a href="https://github.com/fluent/fluentd/pull/3849">#3849</a>, <a href="https://github.com/fluent/fluentd/pull/3866">#3866</a></li> </ul> <p>Enjoy logging!</p> td-agent v4.4.0 has been released https://www.fluentd.org/blog/td-agent-v4.4.0-has-been-released <p>Hi users!</p> <p>We have released td-agent v4.4.0. td-agent is a stable distribution package of Fluentd.</p> <h3>Changes from td-agent v4.3.2</h3> <ul> <li>Update bundled Fluentd from v1.14.6 to v1.15.1. Please see each release announcement of Fluentd for more detail: <ul> <li><a href="fluentd-v1.15.0-has-been-released">Fluentd v1.15.0 has been released</a></li> <li><a href="fluentd-v1.15.1-has-been-released">Fluentd v1.15.1 has been released</a></li> </ul></li> <li>Update OpenSSL to 1.1.1q (for macOS and Windows)</li> <li>Remove <code>TD_AGENT_OPTIONS=</code> from td-agent.service. Instead it&#39;s loaded from /etc/sysconfig/td-agent (rpm) or /etc/default/td-agent (deb).</li> <li>Drop Ubuntu 16.04 &quot;Xenial Xerus&quot; support <a href="td-agent-v4.3.1-has-been-released">as we announced before</a>.</li> <li>Update plugins and gems</li> </ul> <p>Please see <a href="https://github.com/fluent/fluent-package-builder/blob/master/CHANGELOG.md#release-v440---20220729">ChangeLog</a> for more detail.</p> <h3>Download</h3> <p>Please see <a href="/download">the download page</a>.</p> 2022-07-29 <p>Hi users!</p> <p>We have released td-agent v4.4.0. td-agent is a stable distribution package of Fluentd.</p> <h3>Changes from td-agent v4.3.2</h3> <ul> <li>Update bundled Fluentd from v1.14.6 to v1.15.1. Please see each release announcement of Fluentd for more detail: <ul> <li><a href="fluentd-v1.15.0-has-been-released">Fluentd v1.15.0 has been released</a></li> <li><a href="fluentd-v1.15.1-has-been-released">Fluentd v1.15.1 has been released</a></li> </ul></li> <li>Update OpenSSL to 1.1.1q (for macOS and Windows)</li> <li>Remove <code>TD_AGENT_OPTIONS=</code> from td-agent.service. Instead it&#39;s loaded from /etc/sysconfig/td-agent (rpm) or /etc/default/td-agent (deb).</li> <li>Drop Ubuntu 16.04 &quot;Xenial Xerus&quot; support <a href="td-agent-v4.3.1-has-been-released">as we announced before</a>.</li> <li>Update plugins and gems</li> </ul> <p>Please see <a href="https://github.com/fluent/fluent-package-builder/blob/master/CHANGELOG.md#release-v440---20220729">ChangeLog</a> for more detail.</p> <h3>Download</h3> <p>Please see <a href="/download">the download page</a>.</p> Fluentd v1.15.1 has been released https://www.fluentd.org/blog/fluentd-v1.15.1-has-been-released <p>Hi users!</p> <p>We have released v1.15.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1151---20220727">here</a>.</p> <p>This release is a maintenance release of v1.15 series.</p> <h2>Bug Fixes</h2> <h3><code>out_file</code>: Add support for concurrent append</h3> <p>Using <a href="https://docs.fluentd.org/output/file#append">&quot;append&quot; option</a> of <code>out_file</code> in multiple workers could cause file corruption due to write conflicts. This could happen when multiple workers append to the same file at the same time, although it was very rare.</p> <p>In this release, we add a lock-system to Fluentd to prevents write conflicts between multiple workers.</p> <p>This prevents write conflicts of <code>out_file</code>.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3808">#3808</a> for more information.</p> <h2>Miscellaneous bug fixes</h2> <ul> <li><code>in_tail</code>: Show more information on skipping update_watcher. <a href="https://github.com/fluent/fluentd/pull/3829">#3829</a></li> </ul> <p>Enjoy logging!</p> 2022-07-29 <p>Hi users!</p> <p>We have released v1.15.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1151---20220727">here</a>.</p> <p>This release is a maintenance release of v1.15 series.</p> <h2>Bug Fixes</h2> <h3><code>out_file</code>: Add support for concurrent append</h3> <p>Using <a href="https://docs.fluentd.org/output/file#append">&quot;append&quot; option</a> of <code>out_file</code> in multiple workers could cause file corruption due to write conflicts. This could happen when multiple workers append to the same file at the same time, although it was very rare.</p> <p>In this release, we add a lock-system to Fluentd to prevents write conflicts between multiple workers.</p> <p>This prevents write conflicts of <code>out_file</code>.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3808">#3808</a> for more information.</p> <h2>Miscellaneous bug fixes</h2> <ul> <li><code>in_tail</code>: Show more information on skipping update_watcher. <a href="https://github.com/fluent/fluentd/pull/3829">#3829</a></li> </ul> <p>Enjoy logging!</p> Fluentd v1.15.0 has been released https://www.fluentd.org/blog/fluentd-v1.15.0-has-been-released <p>Hi users!</p> <p>We have released v1.15.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1150---20220629">here</a>.</p> <p>This release is a new release of v1.15 series.</p> <p>In this release, we added some new functions and fixed some crash bugs, especially about <code>in_tail</code>.</p> <h2>Enhancement</h2> <h3>Support YAML configuration format</h3> <p>Fluentd now supports YAML configuration format as follows.</p> <pre><code class="yaml">config: - source: $type: forward port: 24224 - source: $type: http port: 9880 - match: $tag: myapp.access $type: file path: /var/log/fluent/access </code></pre> <p>This is not 1-by-1 correspondence for the traditional Fluentd config file syntax.</p> <p>Please see <a href="https://docs.fluentd.org/configuration/config-file-yaml">the document</a> for detailed usage instructions or see <a href="https://github.com/fluent/fluentd/pull/3712">#3712</a> for more information.</p> <h3><code>in_tail</code>: Add log throttling in files based on group rules</h3> <p>In this release, we add a new option directive <code>group</code> to <code>in_tail</code>.</p> <p>From v1.13, We can use <code>read_bytes_limit_per_second</code> to throttle log flow rate, but we can only set it for each <code>source</code> unit.</p> <p>By using this new option, we can throttle log flow rate for each file unit.</p> <p>Please see <a href="https://docs.fluentd.org/input/tail#less-than-group-greater-than-section">the document</a> for detailed usage instructions or see <a href="https://github.com/fluent/fluentd/pull/3535">#3535</a> for more information.</p> <h3>Add <code>restart_worker_interval</code> to system configuration</h3> <p>In this release, we add a new option <code>restart_worker_interval</code>.</p> <p>By default, Fluentd immediately restarts stopped workers, but you may want to keep the worker stopped for a while for some reason. In such cases, you can use this option as follows.</p> <pre><code>&lt;system&gt; restart_worker_interval 1m &lt;/system&gt; </code></pre> <p>By this setting, a stopped worker will restart after 1 minute.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3768">#3768</a> for more information.</p> <h3><code>fluent-ctl</code>: Add a new command <code>dump</code></h3> <p>In this release, we can use a new command <code>dump</code> in <code>fluent-ctl</code>.</p> <p>This command is mainly for Windows.</p> <p>As for non-Windows, you can manually send SIGCONT to each supervisor/worker process to dump their internal status. (<a href="https://docs.fluentd.org/deployment/trouble-shooting#dump-fluentds-internal-information">Dump fluentd&#39;s Internal Information</a>)</p> <p>By using this new command, we can get the internal status on Windows too.</p> <p>On Windows, this makes all Fluentd processes (including all worker processes) dump their internal status to the system temp directory (<code>C:\\Windows\\Temp</code>).</p> <pre><code class="console">$ fluent-ctl dump [PID_OR_SVCNAME] </code></pre> <p>You can omit <code>[PID_OR_SVCNAME]</code> if you run the fluentd as a Windows service, whose service name is the default value <code>fluentdwinsvc</code>.</p> <p>Please see <a href="https://docs.fluentd.org/deployment/command-line-option#about-dump">the document</a> for detailed usage instructions or see <a href="https://github.com/fluent/fluentd/pull/3680">#3680</a> for more information.</p> <h2>Bug Fixes</h2> <ul> <li>Output plugins: Fix a possible crash of flush thread. <a href="https://github.com/fluent/fluentd/pull/3755">#3755</a></li> <li><code>in_tail</code>: Fix crash bugs on Ruby 3.1 on Windows. <a href="https://github.com/fluent/fluentd/pull/3766">#3766</a></li> <li><code>in_tail</code>: Fix a possible crash on file rotation when <code>follow_inodes true</code>. <a href="https://github.com/fluent/fluentd/pull/3754">#3754</a></li> <li><code>in_tail</code>: Fix a bug that in_tail cannot open non-ascii path on Windows. <a href="https://github.com/fluent/fluentd/pull/3774">#3774</a></li> </ul> <h2>Miscellaneous bug fixes</h2> <ul> <li><code>out_forward</code>: Fix to update timeout of cached sockets. <a href="https://github.com/fluent/fluentd/pull/3711">#3711</a></li> <li>Fix a bug that fluentd doesn&#39;t release its own log file even after rotated by external tools. <a href="https://github.com/fluent/fluentd/pull/3782">#3782</a></li> </ul> <p>Enjoy logging!</p> 2022-06-29 <p>Hi users!</p> <p>We have released v1.15.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1150---20220629">here</a>.</p> <p>This release is a new release of v1.15 series.</p> <p>In this release, we added some new functions and fixed some crash bugs, especially about <code>in_tail</code>.</p> <h2>Enhancement</h2> <h3>Support YAML configuration format</h3> <p>Fluentd now supports YAML configuration format as follows.</p> <pre><code class="yaml">config: - source: $type: forward port: 24224 - source: $type: http port: 9880 - match: $tag: myapp.access $type: file path: /var/log/fluent/access </code></pre> <p>This is not 1-by-1 correspondence for the traditional Fluentd config file syntax.</p> <p>Please see <a href="https://docs.fluentd.org/configuration/config-file-yaml">the document</a> for detailed usage instructions or see <a href="https://github.com/fluent/fluentd/pull/3712">#3712</a> for more information.</p> <h3><code>in_tail</code>: Add log throttling in files based on group rules</h3> <p>In this release, we add a new option directive <code>group</code> to <code>in_tail</code>.</p> <p>From v1.13, We can use <code>read_bytes_limit_per_second</code> to throttle log flow rate, but we can only set it for each <code>source</code> unit.</p> <p>By using this new option, we can throttle log flow rate for each file unit.</p> <p>Please see <a href="https://docs.fluentd.org/input/tail#less-than-group-greater-than-section">the document</a> for detailed usage instructions or see <a href="https://github.com/fluent/fluentd/pull/3535">#3535</a> for more information.</p> <h3>Add <code>restart_worker_interval</code> to system configuration</h3> <p>In this release, we add a new option <code>restart_worker_interval</code>.</p> <p>By default, Fluentd immediately restarts stopped workers, but you may want to keep the worker stopped for a while for some reason. In such cases, you can use this option as follows.</p> <pre><code>&lt;system&gt; restart_worker_interval 1m &lt;/system&gt; </code></pre> <p>By this setting, a stopped worker will restart after 1 minute.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/3768">#3768</a> for more information.</p> <h3><code>fluent-ctl</code>: Add a new command <code>dump</code></h3> <p>In this release, we can use a new command <code>dump</code> in <code>fluent-ctl</code>.</p> <p>This command is mainly for Windows.</p> <p>As for non-Windows, you can manually send SIGCONT to each supervisor/worker process to dump their internal status. (<a href="https://docs.fluentd.org/deployment/trouble-shooting#dump-fluentds-internal-information">Dump fluentd&#39;s Internal Information</a>)</p> <p>By using this new command, we can get the internal status on Windows too.</p> <p>On Windows, this makes all Fluentd processes (including all worker processes) dump their internal status to the system temp directory (<code>C:\\Windows\\Temp</code>).</p> <pre><code class="console">$ fluent-ctl dump [PID_OR_SVCNAME] </code></pre> <p>You can omit <code>[PID_OR_SVCNAME]</code> if you run the fluentd as a Windows service, whose service name is the default value <code>fluentdwinsvc</code>.</p> <p>Please see <a href="https://docs.fluentd.org/deployment/command-line-option#about-dump">the document</a> for detailed usage instructions or see <a href="https://github.com/fluent/fluentd/pull/3680">#3680</a> for more information.</p> <h2>Bug Fixes</h2> <ul> <li>Output plugins: Fix a possible crash of flush thread. <a href="https://github.com/fluent/fluentd/pull/3755">#3755</a></li> <li><code>in_tail</code>: Fix crash bugs on Ruby 3.1 on Windows. <a href="https://github.com/fluent/fluentd/pull/3766">#3766</a></li> <li><code>in_tail</code>: Fix a possible crash on file rotation when <code>follow_inodes true</code>. <a href="https://github.com/fluent/fluentd/pull/3754">#3754</a></li> <li><code>in_tail</code>: Fix a bug that in_tail cannot open non-ascii path on Windows. <a href="https://github.com/fluent/fluentd/pull/3774">#3774</a></li> </ul> <h2>Miscellaneous bug fixes</h2> <ul> <li><code>out_forward</code>: Fix to update timeout of cached sockets. <a href="https://github.com/fluent/fluentd/pull/3711">#3711</a></li> <li>Fix a bug that fluentd doesn&#39;t release its own log file even after rotated by external tools. <a href="https://github.com/fluent/fluentd/pull/3782">#3782</a></li> </ul> <p>Enjoy logging!</p>