Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v1.13.3 has been released https://www.fluentd.org/blog/fluentd-v1.13.3-has-been-released <p>Hi users!</p> <p>We have released v1.13.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1133">here</a>.</p> <p>This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about <code>in_tail</code> bugs.</p> <h3><code>in_tail</code>: Fixed a bug that <code>DeletePending</code> state is not cared on Windows</h3> <p>In this release, Windows specific bug was fixed for <code>in_tail</code> plugin.</p> <p>Windows has a concept of <code>DeletePending</code>, which means &quot;DeleteFile() was called on that file. NTFS will remove this file once everyone close the handlers for it&quot;.</p> <p>In the previous versions, it didn&#39;t care about it correctly, so there was a case that deleted file could not be handled properly.</p> <h3><code>in_tail</code>: Fixed a bug that detecting rotation twice on <code>follow_inode</code></h3> <p>In this release, the bug that duplicate events are emitted when <code>follow_inode</code> is <code>true</code> and rotation is occurred.</p> <p>This bug was occurred when the following conditions are met:</p> <ul> <li>Wrong inode is set to <code>TailWatcher</code> when <code>follow_inode</code> is true</li> <li>A key (<code>TargetInfo</code>) in <code>@tails</code> isn&#39;t updated for a same path even if new one has different inode</li> </ul> <p>We recommend to upgrade if you enable <code>follow_inode</code>.</p> <h3><code>in_tail</code>: Fixed some position file related bugs</h3> <p>In this release, some bugs which are related to position file was fixed.</p> <p>Here is the detail of fixed bugs:</p> <ul> <li>When both <code>follow_inodes</code> and <code>pos_file_compaction_interval</code> are specified, file entries that didn&#39;t exist on start up will be deleted unexpectedly when compaction is triggered</li> <li><code>String#bytesize</code> should be used instead of <code>String#size</code> for path length, but it is not used</li> </ul> <p>It causes a bug that file compaction does not work as expected.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.3 </code></pre> <p>Enjoy logging!</p> 2021-07-27 <p>Hi users!</p> <p>We have released v1.13.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1133">here</a>.</p> <p>This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about <code>in_tail</code> bugs.</p> <h3><code>in_tail</code>: Fixed a bug that <code>DeletePending</code> state is not cared on Windows</h3> <p>In this release, Windows specific bug was fixed for <code>in_tail</code> plugin.</p> <p>Windows has a concept of <code>DeletePending</code>, which means &quot;DeleteFile() was called on that file. NTFS will remove this file once everyone close the handlers for it&quot;.</p> <p>In the previous versions, it didn&#39;t care about it correctly, so there was a case that deleted file could not be handled properly.</p> <h3><code>in_tail</code>: Fixed a bug that detecting rotation twice on <code>follow_inode</code></h3> <p>In this release, the bug that duplicate events are emitted when <code>follow_inode</code> is <code>true</code> and rotation is occurred.</p> <p>This bug was occurred when the following conditions are met:</p> <ul> <li>Wrong inode is set to <code>TailWatcher</code> when <code>follow_inode</code> is true</li> <li>A key (<code>TargetInfo</code>) in <code>@tails</code> isn&#39;t updated for a same path even if new one has different inode</li> </ul> <p>We recommend to upgrade if you enable <code>follow_inode</code>.</p> <h3><code>in_tail</code>: Fixed some position file related bugs</h3> <p>In this release, some bugs which are related to position file was fixed.</p> <p>Here is the detail of fixed bugs:</p> <ul> <li>When both <code>follow_inodes</code> and <code>pos_file_compaction_interval</code> are specified, file entries that didn&#39;t exist on start up will be deleted unexpectedly when compaction is triggered</li> <li><code>String#bytesize</code> should be used instead of <code>String#size</code> for path length, but it is not used</li> </ul> <p>It causes a bug that file compaction does not work as expected.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.3 </code></pre> <p>Enjoy logging!</p> Fluentd v1.13.2 has been released https://www.fluentd.org/blog/fluentd-v1.13.2-has-been-released <p>Hi users!</p> <p>We have released v1.13.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1132">here</a>.</p> <p>This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about crash bugs.</p> <h3>Fixed a crash bug during startup phase</h3> <p>In this release, a crash bug was fixed during outputting log at the early stage when parsing invalid config file.</p> <p>This was a bug since v1.13.0. If you use invalid &#39;@&#39; prefix parameter, remove it as a workaround.</p> <h3><code>in_tail</code>: Fixed a bug that remaining lines will be discarded</h3> <p>When rotation is occurred with the throttling feature is enabled, there is a case that remaining lines will be discarded.</p> <h3>Added support to customize configuration of oj options</h3> <p>In this release, it is able to customize OJ options via environment variables.</p> <p>Use the following environment variables to customize behavior of OJ.</p> <ul> <li><code>FLUENT_OJ_OPTION_BIGDECIMAL_LOAD</code> (float by default)</li> <li><code>FLUENT_OJ_OPTION_MAX_NESTING</code></li> <li><code>FLUENT_OJ_OPTION_MODE</code> (compat by default)</li> <li><code>FLUENT_OJ_OPTION_USE_TO_JSON</code> (true by default)</li> </ul> <p>See <a href="https://github.com/ohler55/oj/blob/develop/pages/Options.md">Oj Options</a> about details of oj options.</p> <h3>Generating template of storage plugin was supported</h3> <p>Since v1.13.2, <code>fluent-plugin-generate</code> command has been supported to generate template files for storage plugin.</p> <pre><code>$ fluent-plugin-generate storage foo License: Apache-2.0 create Gemfile create README.md create Rakefile create fluent-plugin-foo.gemspec create lib/fluent/plugin/storage_foo.rb create test/helper.rb create test/plugin/test_storage_foo.rb Initialized empty Git repository in /work/fluentd/fluentd/fluentd/fluent-plugin-foo/.git/ </code></pre> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.2 </code></pre> <p>Enjoy logging!</p> 2021-07-12 <p>Hi users!</p> <p>We have released v1.13.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1132">here</a>.</p> <p>This release is a maintenance release of v1.13 series. We recommend to upgrade Fluentd because it contains fixes about crash bugs.</p> <h3>Fixed a crash bug during startup phase</h3> <p>In this release, a crash bug was fixed during outputting log at the early stage when parsing invalid config file.</p> <p>This was a bug since v1.13.0. If you use invalid &#39;@&#39; prefix parameter, remove it as a workaround.</p> <h3><code>in_tail</code>: Fixed a bug that remaining lines will be discarded</h3> <p>When rotation is occurred with the throttling feature is enabled, there is a case that remaining lines will be discarded.</p> <h3>Added support to customize configuration of oj options</h3> <p>In this release, it is able to customize OJ options via environment variables.</p> <p>Use the following environment variables to customize behavior of OJ.</p> <ul> <li><code>FLUENT_OJ_OPTION_BIGDECIMAL_LOAD</code> (float by default)</li> <li><code>FLUENT_OJ_OPTION_MAX_NESTING</code></li> <li><code>FLUENT_OJ_OPTION_MODE</code> (compat by default)</li> <li><code>FLUENT_OJ_OPTION_USE_TO_JSON</code> (true by default)</li> </ul> <p>See <a href="https://github.com/ohler55/oj/blob/develop/pages/Options.md">Oj Options</a> about details of oj options.</p> <h3>Generating template of storage plugin was supported</h3> <p>Since v1.13.2, <code>fluent-plugin-generate</code> command has been supported to generate template files for storage plugin.</p> <pre><code>$ fluent-plugin-generate storage foo License: Apache-2.0 create Gemfile create README.md create Rakefile create fluent-plugin-foo.gemspec create lib/fluent/plugin/storage_foo.rb create test/helper.rb create test/plugin/test_storage_foo.rb Initialized empty Git repository in /work/fluentd/fluentd/fluentd/fluent-plugin-foo/.git/ </code></pre> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.2 </code></pre> <p>Enjoy logging!</p> Fluentd v1.13.1 has been released https://www.fluentd.org/blog/fluentd-v1.13.1-has-been-released <p>Hi users!</p> <p>We have released v1.13.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1131">here</a>.</p> <p>This release is a maintenance release of v1.13 series.</p> <h3><code>in_tail</code>: Fixed to remove too much verbose debugging logs</h3> <p>It was unexpectedly introduced by <a href="https://github.com/fluent/fluentd/pull/3185">#3185</a> - log throttling feature in v1.13.0. In this release, too much verbose debug logging was removed.</p> <p>It has nothing to do with it when you didn&#39;t use <code>in_tail</code> with debug level.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4 or later. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.1 </code></pre> <p>Enjoy logging!</p> 2021-06-25 <p>Hi users!</p> <p>We have released v1.13.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v1131">here</a>.</p> <p>This release is a maintenance release of v1.13 series.</p> <h3><code>in_tail</code>: Fixed to remove too much verbose debugging logs</h3> <p>It was unexpectedly introduced by <a href="https://github.com/fluent/fluentd/pull/3185">#3185</a> - log throttling feature in v1.13.0. In this release, too much verbose debug logging was removed.</p> <p>It has nothing to do with it when you didn&#39;t use <code>in_tail</code> with debug level.</p> <h3>Upgrade attention for TD Agent users</h3> <p>If you use Fluentd v1.12 or later, we recommend to use at least v1.12.4 or later. It is because <code>in_tail</code> contains serious bugs in it.</p> <p>Note that even latest td-agent 4.1.1 also ships a buggy Fluentd v1.12.3. You should upgrade bundled Fluentd by yourself. This issue will be fixed in the future release of td-agent.</p> <pre><code>$ sudo td-agent-gem install fluentd --version=1.13.1 </code></pre> <p>Enjoy logging!</p> Fluentd v1.13.0 has been released https://www.fluentd.org/blog/fluentd-v1.13.0-has-been-released <p>Hi users!</p> <p>We have released v1.13.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v113">here</a>.</p> <p>There are some topics in this release.</p> <ul> <li><code>service_discovery</code>: extended to support service discovery manager in simpler way</li> <li><code>in_tail</code>: handling log throttling per file feature has been supported</li> <li><code>in_http</code>: HTTP GET requests has been supported</li> <li>The log rotate settings in system configuration has been supported</li> <li><code>fluent-cat</code>: the issue resending secondary file in specific format has been fixed</li> </ul> <p>There are many <code>in_tail</code> bug fixes, we recommend to upgrade to v1.13.0.</p> <h3><code>service_discovery</code>: extended to support service discovery manager in simpler way</h3> <p>In the previous versions, if plugin author supports <a href="https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-service_discovery">service_discovery</a>, it must build <code>configurations</code> structure explicitly, but it is complicated way.</p> <p>Since Fluentd v1.13.0, it is enough to call <code>#service_discovery_configure</code> from <code>Plugin#configure</code>. It makes implementation costs lower.</p> <h3><code>in_tail</code>: handling log throttling per file feature has been supported</h3> <p>In the previous versions, there is no way to limit log throttling.</p> <p>Since Fluentd v1.13.0, the configuration parameter - <code>read_bytes_limit_per_second</code> is introduced.</p> <pre><code>&lt;source&gt; @type tail ... read_bytes_limit_per_second 4k &lt;/source&gt; </code></pre> <p>As <code>read_bytes_limit_per_second</code> accepts numbers with SI prefix such as <code>4k</code>.</p> <h3><code>in_http</code>: HTTP GET requests has been supported</h3> <p>This functionality is implemented to support health check feature on Azure App.</p> <p>Azure App uses GET requests to check if the HTTP server is working all right. In the previous versions of Fluentd, it responds with &quot;400 Bad Requests&quot; to HTTP GET, so it does not work well for health checking purpose.</p> <p>Since Fluentd v1.13.0, Fluentd returns a &quot;200 OK&quot;.</p> <p>Thanks Josh Keegan for requesting feature request!</p> <h3>The log rotate settings in system configuration has been supported</h3> <p>In the previous versions, log rotation options were supported as <code>--log-rotate-age</code> or <code>--log-rotate-size</code> via command line options.</p> <p>On Windows, as Fluentd is launched as a windows service, it is required to configure again via <code>--reg-winsvc-fluentdopt</code> or edit <code>fluentdopt</code> registry key for log rotate customization.</p> <p>These approach are not convenient for Windows users, so it may be better to support more comprehensive solution - customize via configuration file like this:</p> <pre><code>&lt;system&gt; &lt;log&gt; rotate_age 5 rotate_size 1048576 &lt;/log&gt; &lt;/system&gt; </code></pre> <p>Now you can customize via <code>rotate_age</code> and <code>rotate_size</code> in <code>log</code> section.</p> <h3><code>fluent-cat</code>: the issue resending secondary file in specific format has been fixed</h3> <p>The way to resend secondary is explained as <a href="https://docs.fluentd.org/output/secondary_file#how-to-resend-secondary-file">How to resend secondary file</a>, but it does not work if the record contains <code>Fluent::EventTime</code>. It raises <code>Input must be a map (got Array)</code> error.</p> <p>In this release, it supports such a secondary file too.</p> <p>Note that <code>fluent-cat</code> ignore timestamp which is stored in secondary records. (It is intended behavior for keeping consistency)</p> <h3>Miscellaneous bug fixes</h3> <ul> <li>Fixed to disable <code>trace_instruction</code> when <code>RubyVM::InstructionSequence</code> is available. It improves compatibility with <code>truffleruby</code> some extent. <a href="https://github.com/fluent/fluentd/pull/3376">#3376</a></li> <li><code>in_tail</code>: Safely skip files which are used by another process on Windows. It improves exception handling about <code>ERROR_SHARING_VIOLATION</code> on Windows. <a href="https://github.com/fluent/fluentd/pull/3378">#3378</a></li> </ul> <h3>Community topic</h3> <p>About three months has passed since community forum was moved to <a href="https://discuss.fluentd.org/">Discuss</a>. <a href="https://discuss.fluentd.org/c/fluentd-help/5">Fluentd category</a> is used actively nowadays! Note that fluentd forum on google groups is already deprecated.</p> <p>Enjoy logging!</p> 2021-05-29 <p>Hi users!</p> <p>We have released v1.13.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#v113">here</a>.</p> <p>There are some topics in this release.</p> <ul> <li><code>service_discovery</code>: extended to support service discovery manager in simpler way</li> <li><code>in_tail</code>: handling log throttling per file feature has been supported</li> <li><code>in_http</code>: HTTP GET requests has been supported</li> <li>The log rotate settings in system configuration has been supported</li> <li><code>fluent-cat</code>: the issue resending secondary file in specific format has been fixed</li> </ul> <p>There are many <code>in_tail</code> bug fixes, we recommend to upgrade to v1.13.0.</p> <h3><code>service_discovery</code>: extended to support service discovery manager in simpler way</h3> <p>In the previous versions, if plugin author supports <a href="https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-service_discovery">service_discovery</a>, it must build <code>configurations</code> structure explicitly, but it is complicated way.</p> <p>Since Fluentd v1.13.0, it is enough to call <code>#service_discovery_configure</code> from <code>Plugin#configure</code>. It makes implementation costs lower.</p> <h3><code>in_tail</code>: handling log throttling per file feature has been supported</h3> <p>In the previous versions, there is no way to limit log throttling.</p> <p>Since Fluentd v1.13.0, the configuration parameter - <code>read_bytes_limit_per_second</code> is introduced.</p> <pre><code>&lt;source&gt; @type tail ... read_bytes_limit_per_second 4k &lt;/source&gt; </code></pre> <p>As <code>read_bytes_limit_per_second</code> accepts numbers with SI prefix such as <code>4k</code>.</p> <h3><code>in_http</code>: HTTP GET requests has been supported</h3> <p>This functionality is implemented to support health check feature on Azure App.</p> <p>Azure App uses GET requests to check if the HTTP server is working all right. In the previous versions of Fluentd, it responds with &quot;400 Bad Requests&quot; to HTTP GET, so it does not work well for health checking purpose.</p> <p>Since Fluentd v1.13.0, Fluentd returns a &quot;200 OK&quot;.</p> <p>Thanks Josh Keegan for requesting feature request!</p> <h3>The log rotate settings in system configuration has been supported</h3> <p>In the previous versions, log rotation options were supported as <code>--log-rotate-age</code> or <code>--log-rotate-size</code> via command line options.</p> <p>On Windows, as Fluentd is launched as a windows service, it is required to configure again via <code>--reg-winsvc-fluentdopt</code> or edit <code>fluentdopt</code> registry key for log rotate customization.</p> <p>These approach are not convenient for Windows users, so it may be better to support more comprehensive solution - customize via configuration file like this:</p> <pre><code>&lt;system&gt; &lt;log&gt; rotate_age 5 rotate_size 1048576 &lt;/log&gt; &lt;/system&gt; </code></pre> <p>Now you can customize via <code>rotate_age</code> and <code>rotate_size</code> in <code>log</code> section.</p> <h3><code>fluent-cat</code>: the issue resending secondary file in specific format has been fixed</h3> <p>The way to resend secondary is explained as <a href="https://docs.fluentd.org/output/secondary_file#how-to-resend-secondary-file">How to resend secondary file</a>, but it does not work if the record contains <code>Fluent::EventTime</code>. It raises <code>Input must be a map (got Array)</code> error.</p> <p>In this release, it supports such a secondary file too.</p> <p>Note that <code>fluent-cat</code> ignore timestamp which is stored in secondary records. (It is intended behavior for keeping consistency)</p> <h3>Miscellaneous bug fixes</h3> <ul> <li>Fixed to disable <code>trace_instruction</code> when <code>RubyVM::InstructionSequence</code> is available. It improves compatibility with <code>truffleruby</code> some extent. <a href="https://github.com/fluent/fluentd/pull/3376">#3376</a></li> <li><code>in_tail</code>: Safely skip files which are used by another process on Windows. It improves exception handling about <code>ERROR_SHARING_VIOLATION</code> on Windows. <a href="https://github.com/fluent/fluentd/pull/3378">#3378</a></li> </ul> <h3>Community topic</h3> <p>About three months has passed since community forum was moved to <a href="https://discuss.fluentd.org/">Discuss</a>. <a href="https://discuss.fluentd.org/c/fluentd-help/5">Fluentd category</a> is used actively nowadays! Note that fluentd forum on google groups is already deprecated.</p> <p>Enjoy logging!</p> Fluentd v1.12.4 has been released https://www.fluentd.org/blog/fluentd-v1.12.4-has-been-released <p>Hi users!</p> <p>We have released v1.12.4. This release is a maintenance release of v1.12 series. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/v1.12.4/CHANGELOG.md#release-v1124---20210526">here</a>.</p> <h3><code>in_tail</code>: Fix a bug that refresh_watcher fails to handle file rotations</h3> <p>In the previous Fluentd v1.11.x implementation uses path based tailing keys. Since v1.12, <code>follow_inodes</code> feature is introduced, but as a side effects unexpectedly, refresh_watcher tend to fail to handle file rotations correctly.</p> <p>This version contains a fix for the issue only. <a href="https://github.com/fluent/fluentd/pull/3393">#3393</a></p> <p>Enjoy logging!</p> 2021-05-26 <p>Hi users!</p> <p>We have released v1.12.4. This release is a maintenance release of v1.12 series. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/v1.12.4/CHANGELOG.md#release-v1124---20210526">here</a>.</p> <h3><code>in_tail</code>: Fix a bug that refresh_watcher fails to handle file rotations</h3> <p>In the previous Fluentd v1.11.x implementation uses path based tailing keys. Since v1.12, <code>follow_inodes</code> feature is introduced, but as a side effects unexpectedly, refresh_watcher tend to fail to handle file rotations correctly.</p> <p>This version contains a fix for the issue only. <a href="https://github.com/fluent/fluentd/pull/3393">#3393</a></p> <p>Enjoy logging!</p>