Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v1.14.1 has been released https://www.fluentd.org/blog/fluentd-v1.14.1-has-been-released <p>Hi users!</p> <p>We have released v1.14.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1141">here</a>.</p> <p>This release is a maintenance release of v1.14 series.</p> <p>We recommend to upgrade Fluentd because it contains fixes about <code>in_tail</code> crash bug.</p> <h3>in_tail: Fixed a crash bug that it raise undefined method of eof? error</h3> <p>In this release, one of <code>in_tail</code> plugin bug was fixed.</p> <p>This crash bug will happen only when <code>read_bytes_limit_per_second</code> was specified. <code>read_bytes_limit_per_second</code> parameter was introduced since Fluentd v1.13.0, so it affects v1.13.0 or later with <code>read_bytes_limit_per_second</code> was specified.</p> <p>If you don&#39;t use <code>read_bytes_limit_per_second</code> configuration parameter at all, it has nothing to do with it.</p> <h3>Fixed a error when using <code>@include</code> directive</h3> <p>If you use the following example, it causes an unexpected error.</p> <pre><code>@include &quot;https://raw.githubusercontent.com/fluent/fluentd/master/example/in_forward.conf&quot; </code></pre> <p>The error occurs when http or https scheme URI is used in <code>@include</code> directive with Ruby 3. On Ruby 2.7, it raise an warning message (using deprecated method), not an error.</p> <h3>out_copy: Fixed to suppress a wrong warning for ignore_if_prev_success</h3> <p>Because of internal inappropriate conditional handling, even though a user set <code>ignore_if_prev_success</code>, it doesn&#39;t work as expected.</p> <p>Instead, it emits the following warning.</p> <pre><code>2021-09-24 14:10:54 +0900 [warn]: #0 ignore_errors are specified in all &lt;store&gt;, but ignore_if_prev_success is not specified. Is this intended? </code></pre> <p>This behavior was fixed.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>As latest td-agent 4.2.0 bundles Fluentd 1.13.3. If you want to upgrade Fluentd further more, upgrade it by yourself.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.14.1 </code></pre> <p>Enjoy logging!</p> 2021-09-29 <p>Hi users!</p> <p>We have released v1.14.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1141">here</a>.</p> <p>This release is a maintenance release of v1.14 series.</p> <p>We recommend to upgrade Fluentd because it contains fixes about <code>in_tail</code> crash bug.</p> <h3>in_tail: Fixed a crash bug that it raise undefined method of eof? error</h3> <p>In this release, one of <code>in_tail</code> plugin bug was fixed.</p> <p>This crash bug will happen only when <code>read_bytes_limit_per_second</code> was specified. <code>read_bytes_limit_per_second</code> parameter was introduced since Fluentd v1.13.0, so it affects v1.13.0 or later with <code>read_bytes_limit_per_second</code> was specified.</p> <p>If you don&#39;t use <code>read_bytes_limit_per_second</code> configuration parameter at all, it has nothing to do with it.</p> <h3>Fixed a error when using <code>@include</code> directive</h3> <p>If you use the following example, it causes an unexpected error.</p> <pre><code>@include &quot;https://raw.githubusercontent.com/fluent/fluentd/master/example/in_forward.conf&quot; </code></pre> <p>The error occurs when http or https scheme URI is used in <code>@include</code> directive with Ruby 3. On Ruby 2.7, it raise an warning message (using deprecated method), not an error.</p> <h3>out_copy: Fixed to suppress a wrong warning for ignore_if_prev_success</h3> <p>Because of internal inappropriate conditional handling, even though a user set <code>ignore_if_prev_success</code>, it doesn&#39;t work as expected.</p> <p>Instead, it emits the following warning.</p> <pre><code>2021-09-24 14:10:54 +0900 [warn]: #0 ignore_errors are specified in all &lt;store&gt;, but ignore_if_prev_success is not specified. Is this intended? </code></pre> <p>This behavior was fixed.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>As latest td-agent 4.2.0 bundles Fluentd 1.13.3. If you want to upgrade Fluentd further more, upgrade it by yourself.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.14.1 </code></pre> <p>Enjoy logging!</p> Fluentd v1.14.0 has been released https://www.fluentd.org/blog/fluentd-v1.14.0-has-been-released <p>Hi users!</p> <p>We have released v1.14.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1140">here</a>.</p> <p>This release is a new release of v1.14 series. We recommend to upgrade Fluentd because it contains fixes about <code>in_tail</code> bugs because a bug that deleted paths are not removed was fixed.</p> <h3>System configuration about metrics parameters were added</h3> <p>In this release, new system configuration parameters were added.</p> <ul> <li><code>enable_input_metrics</code></li> <li><code>enable_size_metrics</code></li> </ul> <p>You can enable these parameters in <code>&lt;system&gt;</code> directive.</p> <pre><code>&lt;system&gt; enable_input_metrics enable_size_metrics &lt;/system&gt; </code></pre> <p>This feature might need to pay higher CPU cost, so these event metrics features are disabled by default. These features are also enabled by <code>--enable-input-metrics</code>,<code>--enable-size-metrics</code> command line.</p> <p>And more, metrics plugin interface was improved. This improvements are introduced to enhance collecting metrics functionality.</p> <p>In the future, 3rd party plugin will be able to handle these metrics mechanism.</p> <h3><code>@ROOT</code> label was changed as reserved label</h3> <p>In this release, the <code>@ROOT</code> label is reserved for a builtin label which is used for getting root router by plugin&#39;s <code>event_emitter_router</code> API.</p> <p>This is incompatible change. Do not use <code>@ROOT</code> like <code>&lt;label @ROOT&gt;</code>.</p> <p>This label is aimed to assign a label back to the default route. For example, timed-out event records are handled by the concat filter which can be sent to the default route.</p> <h3><code>in_syslog</code>: <code>send_keepalive_packet</code> option was added</h3> <p>TCP keepalive was supported by <code>in_syslog</code> to avoid dead connection, as same as <code>in_forward</code>.</p> <h3><code>in_http</code>: <code>cors_allow_credentials</code> option was added</h3> <p>This option tells browsers whether to expose the response to frontend when the credentials mode is <code>include</code>.</p> <p>The default value is false.</p> <h3>Fixed a crash bug on logging peer host name with errors</h3> <p>When a client with invalid TLS certificates is trying to connect, it is hard to investigate which client has the problem without peer information, so it was changed to log peer information since v1.12.4.</p> <p>But, if it fails to resolve peer name information when logging, it caused a crash bug unexpectedly. In this release, this bug was fixed.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that the previous version of td-agent 4.1.1 ships a buggy Fluentd v1.12.3. We recommend upgrading to latest td-agent 4.2.0. It bundles Fluentd 1.13.3. If you want to upgrade Fluentd further more, upgrade it by yourself.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.14.0 </code></pre> <p>Enjoy logging!</p> 2021-08-30 <p>Hi users!</p> <p>We have released v1.14.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1140">here</a>.</p> <p>This release is a new release of v1.14 series. We recommend to upgrade Fluentd because it contains fixes about <code>in_tail</code> bugs because a bug that deleted paths are not removed was fixed.</p> <h3>System configuration about metrics parameters were added</h3> <p>In this release, new system configuration parameters were added.</p> <ul> <li><code>enable_input_metrics</code></li> <li><code>enable_size_metrics</code></li> </ul> <p>You can enable these parameters in <code>&lt;system&gt;</code> directive.</p> <pre><code>&lt;system&gt; enable_input_metrics enable_size_metrics &lt;/system&gt; </code></pre> <p>This feature might need to pay higher CPU cost, so these event metrics features are disabled by default. These features are also enabled by <code>--enable-input-metrics</code>,<code>--enable-size-metrics</code> command line.</p> <p>And more, metrics plugin interface was improved. This improvements are introduced to enhance collecting metrics functionality.</p> <p>In the future, 3rd party plugin will be able to handle these metrics mechanism.</p> <h3><code>@ROOT</code> label was changed as reserved label</h3> <p>In this release, the <code>@ROOT</code> label is reserved for a builtin label which is used for getting root router by plugin&#39;s <code>event_emitter_router</code> API.</p> <p>This is incompatible change. Do not use <code>@ROOT</code> like <code>&lt;label @ROOT&gt;</code>.</p> <p>This label is aimed to assign a label back to the default route. For example, timed-out event records are handled by the concat filter which can be sent to the default route.</p> <h3><code>in_syslog</code>: <code>send_keepalive_packet</code> option was added</h3> <p>TCP keepalive was supported by <code>in_syslog</code> to avoid dead connection, as same as <code>in_forward</code>.</p> <h3><code>in_http</code>: <code>cors_allow_credentials</code> option was added</h3> <p>This option tells browsers whether to expose the response to frontend when the credentials mode is <code>include</code>.</p> <p>The default value is false.</p> <h3>Fixed a crash bug on logging peer host name with errors</h3> <p>When a client with invalid TLS certificates is trying to connect, it is hard to investigate which client has the problem without peer information, so it was changed to log peer information since v1.12.4.</p> <p>But, if it fails to resolve peer name information when logging, it caused a crash bug unexpectedly. In this release, this bug was fixed.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that the previous version of td-agent 4.1.1 ships a buggy Fluentd v1.12.3. We recommend upgrading to latest td-agent 4.2.0. It bundles Fluentd 1.13.3. If you want to upgrade Fluentd further more, upgrade it by yourself.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.14.0 </code></pre> <p>Enjoy logging!</p> TD Agent v4.2.0 has been released https://www.fluentd.org/blog/td-agent-v4.2.0-has-been-released <p>Hi users!</p> <p>We have released v4.2.0. ChangeLog is <a href="https://github.com/fluent-plugins-nursery/td-agent-builder/blob/master/CHANGELOG.md#release-v420---20210729">here</a>.</p> <p>This release is a maintenance release of v4.x series. We recommend to upgrade TD Agent because it contains latest Fluentd v1.13.3.</p> <h3>Jemalloc was downgraded to v3.6.0</h3> <p>Because of excessive amounts of memory consuming issue, bundled version of jemalloc was downgraded from v5.2.1 to v3.6.0.</p> <p>This issue seems a well known issue as follows:</p> <ul> <li><a href="https://bugs.ruby-lang.org/issues/14718">https://bugs.ruby-lang.org/issues/14718</a></li> <li><a href="https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html">https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html</a></li> <li><a href="https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html">https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html</a></li> </ul> <p>Jemalloc v3.6.0 is stable enough (according to jemlloc&#39;s web site, it still seems be supported)</p> <p>See <a href="https://github.com/fluent-plugins-nursery/td-agent-builder/issues/305">https://github.com/fluent-plugins-nursery/td-agent-builder/issues/305</a> about details.</p> <h3>Dependency to perl package was removed</h3> <p>In the previous versions, <code>td-agent</code> has dependency to perl packages. This is because it bundles <code>jeprof</code>, but users do not want to install extra dependencies. So, we decided to drop it.</p> <p>Enjoy logging!</p> 2021-08-12 <p>Hi users!</p> <p>We have released v4.2.0. ChangeLog is <a href="https://github.com/fluent-plugins-nursery/td-agent-builder/blob/master/CHANGELOG.md#release-v420---20210729">here</a>.</p> <p>This release is a maintenance release of v4.x series. We recommend to upgrade TD Agent because it contains latest Fluentd v1.13.3.</p> <h3>Jemalloc was downgraded to v3.6.0</h3> <p>Because of excessive amounts of memory consuming issue, bundled version of jemalloc was downgraded from v5.2.1 to v3.6.0.</p> <p>This issue seems a well known issue as follows:</p> <ul> <li><a href="https://bugs.ruby-lang.org/issues/14718">https://bugs.ruby-lang.org/issues/14718</a></li> <li><a href="https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html">https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html</a></li> <li><a href="https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html">https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html</a></li> </ul> <p>Jemalloc v3.6.0 is stable enough (according to jemlloc&#39;s web site, it still seems be supported)</p> <p>See <a href="https://github.com/fluent-plugins-nursery/td-agent-builder/issues/305">https://github.com/fluent-plugins-nursery/td-agent-builder/issues/305</a> about details.</p> <h3>Dependency to perl package was removed</h3> <p>In the previous versions, <code>td-agent</code> has dependency to perl packages. This is because it bundles <code>jeprof</code>, but users do not want to install extra dependencies. So, we decided to drop it.</p> <p>Enjoy logging!</p> Fluentd v1.13.3 has been released https://www.fluentd.org/blog/fluentd-v1.13.3-has-been-released <p>Hi users!</p> <p>We have released v1.13.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1133">here</a>.</p> <p>This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about <code>in_tail</code> bugs.</p> <h3><code>in_tail</code>: Fixed a bug that <code>DeletePending</code> state is not cared on Windows</h3> <p>In this release, Windows specific bug was fixed for <code>in_tail</code> plugin.</p> <p>Windows has a concept of <code>DeletePending</code>, which means &quot;DeleteFile() was called on that file. NTFS will remove this file once everyone close the handlers for it&quot;.</p> <p>In the previous versions, it didn&#39;t care about it correctly, so there was a case that deleted file could not be handled properly.</p> <h3><code>in_tail</code>: Fixed a bug that detecting rotation twice on <code>follow_inode</code></h3> <p>In this release, the bug that duplicate events are emitted when <code>follow_inode</code> is <code>true</code> and rotation is occurred.</p> <p>This bug was occurred when the following conditions are met:</p> <ul> <li>Wrong inode is set to <code>TailWatcher</code> when <code>follow_inode</code> is true</li> <li>A key (<code>TargetInfo</code>) in <code>@tails</code> isn&#39;t updated for a same path even if new one has different inode</li> </ul> <p>We recommend to upgrade if you enable <code>follow_inode</code>.</p> <h3><code>in_tail</code>: Fixed some position file related bugs</h3> <p>In this release, some bugs which are related to position file was fixed.</p> <p>Here is the detail of fixed bugs:</p> <ul> <li>When both <code>follow_inodes</code> and <code>pos_file_compaction_interval</code> are specified, file entries that didn&#39;t exist on start up will be deleted unexpectedly when compaction is triggered</li> <li><code>String#bytesize</code> should be used instead of <code>String#size</code> for path length, but it is not used</li> </ul> <p>It causes a bug that file compaction does not work as expected.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.3 </code></pre> <p>Enjoy logging!</p> 2021-07-27 <p>Hi users!</p> <p>We have released v1.13.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1133">here</a>.</p> <p>This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about <code>in_tail</code> bugs.</p> <h3><code>in_tail</code>: Fixed a bug that <code>DeletePending</code> state is not cared on Windows</h3> <p>In this release, Windows specific bug was fixed for <code>in_tail</code> plugin.</p> <p>Windows has a concept of <code>DeletePending</code>, which means &quot;DeleteFile() was called on that file. NTFS will remove this file once everyone close the handlers for it&quot;.</p> <p>In the previous versions, it didn&#39;t care about it correctly, so there was a case that deleted file could not be handled properly.</p> <h3><code>in_tail</code>: Fixed a bug that detecting rotation twice on <code>follow_inode</code></h3> <p>In this release, the bug that duplicate events are emitted when <code>follow_inode</code> is <code>true</code> and rotation is occurred.</p> <p>This bug was occurred when the following conditions are met:</p> <ul> <li>Wrong inode is set to <code>TailWatcher</code> when <code>follow_inode</code> is true</li> <li>A key (<code>TargetInfo</code>) in <code>@tails</code> isn&#39;t updated for a same path even if new one has different inode</li> </ul> <p>We recommend to upgrade if you enable <code>follow_inode</code>.</p> <h3><code>in_tail</code>: Fixed some position file related bugs</h3> <p>In this release, some bugs which are related to position file was fixed.</p> <p>Here is the detail of fixed bugs:</p> <ul> <li>When both <code>follow_inodes</code> and <code>pos_file_compaction_interval</code> are specified, file entries that didn&#39;t exist on start up will be deleted unexpectedly when compaction is triggered</li> <li><code>String#bytesize</code> should be used instead of <code>String#size</code> for path length, but it is not used</li> </ul> <p>It causes a bug that file compaction does not work as expected.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.3 </code></pre> <p>Enjoy logging!</p> Fluentd v1.13.2 has been released https://www.fluentd.org/blog/fluentd-v1.13.2-has-been-released <p>Hi users!</p> <p>We have released v1.13.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1132">here</a>.</p> <p>This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about crash bugs.</p> <h3>Fixed a crash bug during startup phase</h3> <p>In this release, a crash bug was fixed during outputting log at the early stage when parsing invalid config file.</p> <p>This was a bug since v1.13.0. If you use invalid &#39;@&#39; prefix parameter, remove it as a workaround.</p> <h3><code>in_tail</code>: Fixed a bug that remaining lines will be discarded</h3> <p>When rotation is occurred with the throttling feature is enabled, there is a case that remaining lines will be discarded.</p> <h3>Added support to customize configuration of oj options</h3> <p>In this release, it is able to customize OJ options via environment variables.</p> <p>Use the following environment variables to customize behavior of OJ.</p> <ul> <li><code>FLUENT_OJ_OPTION_BIGDECIMAL_LOAD</code> (float by default)</li> <li><code>FLUENT_OJ_OPTION_MAX_NESTING</code></li> <li><code>FLUENT_OJ_OPTION_MODE</code> (compat by default)</li> <li><code>FLUENT_OJ_OPTION_USE_TO_JSON</code> (true by default)</li> </ul> <p>See <a href="https://github.com/ohler55/oj/blob/develop/pages/Options.md">Oj Options</a> about details of oj options.</p> <h3>Generating template of storage plugin was supported</h3> <p>Since v1.13.2, <code>fluent-plugin-generate</code> command has been supported to generate template files for storage plugin.</p> <pre><code>$ fluent-plugin-generate storage foo License: Apache-2.0 create Gemfile create README.md create Rakefile create fluent-plugin-foo.gemspec create lib/fluent/plugin/storage_foo.rb create test/helper.rb create test/plugin/test_storage_foo.rb Initialized empty Git repository in /work/fluentd/fluentd/fluentd/fluent-plugin-foo/.git/ </code></pre> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.2 </code></pre> <p>Enjoy logging!</p> 2021-07-12 <p>Hi users!</p> <p>We have released v1.13.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1132">here</a>.</p> <p>This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about crash bugs.</p> <h3>Fixed a crash bug during startup phase</h3> <p>In this release, a crash bug was fixed during outputting log at the early stage when parsing invalid config file.</p> <p>This was a bug since v1.13.0. If you use invalid &#39;@&#39; prefix parameter, remove it as a workaround.</p> <h3><code>in_tail</code>: Fixed a bug that remaining lines will be discarded</h3> <p>When rotation is occurred with the throttling feature is enabled, there is a case that remaining lines will be discarded.</p> <h3>Added support to customize configuration of oj options</h3> <p>In this release, it is able to customize OJ options via environment variables.</p> <p>Use the following environment variables to customize behavior of OJ.</p> <ul> <li><code>FLUENT_OJ_OPTION_BIGDECIMAL_LOAD</code> (float by default)</li> <li><code>FLUENT_OJ_OPTION_MAX_NESTING</code></li> <li><code>FLUENT_OJ_OPTION_MODE</code> (compat by default)</li> <li><code>FLUENT_OJ_OPTION_USE_TO_JSON</code> (true by default)</li> </ul> <p>See <a href="https://github.com/ohler55/oj/blob/develop/pages/Options.md">Oj Options</a> about details of oj options.</p> <h3>Generating template of storage plugin was supported</h3> <p>Since v1.13.2, <code>fluent-plugin-generate</code> command has been supported to generate template files for storage plugin.</p> <pre><code>$ fluent-plugin-generate storage foo License: Apache-2.0 create Gemfile create README.md create Rakefile create fluent-plugin-foo.gemspec create lib/fluent/plugin/storage_foo.rb create test/helper.rb create test/plugin/test_storage_foo.rb Initialized empty Git repository in /work/fluentd/fluentd/fluentd/fluent-plugin-foo/.git/ </code></pre> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.2 </code></pre> <p>Enjoy logging!</p>