Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More fluent-package v5.0.4 has been released https://www.fluentd.org/blog/fluent-package-v5.0.4-has-been-released <p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.4">v5.0.4</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.4 and a foolproof mechanism was implemented to prevent launching duplicated Fluentd instances, we recommend upgrading to fluent-package v5.0.4!</p> <h3>Changes from fluent-package v5.0.3</h3> <ul> <li>Update ruby to 3.2.4 (<a href="https://github.com/fluent/fluent-package-builder/pull/645">#645</a>)</li> <li>Fixed to prevent launching Fluentd wrongly if the service is already running (<a href="https://github.com/fluent/fluent-package-builder/pull/648">#648</a>,<a href="https://github.com/fluent/fluent-package-builder/pull/649">#649</a>)</li> <li>Added support for Ubuntu 24.04 (Noble Numbat)</li> </ul> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b> In this release, dropped CentOS 7 package because of EOL. </div> <h3>Fixed to prevent launching Fluentd wrongly if the service is already running</h3> <p>In this release, a foolproof mechanism was implemented to prevent launching Fluentd wrongly if the service is already running.</p> <p>As you know, you can check the version of Fluentd with <code>fluentd --version</code>, but there is a case that <code>fluentd -v</code> is executed wrongly to do it.</p> <p>When already running Fluentd as a service, <code>fluentd -v</code> launches a duplicated Fluentd instance with the same fluentd configuration. If you launch duplicated Fluentd instances, it causes the corruption of processing Fluentd buffer. To prevent such a situation, a foolproof was implemented now.</p> <p>For example, if Fluentd is running as a service, launching Fluentd causes an error to block it.</p> <p>Here is the example on Windows:</p> <pre><code>&gt; fluentd Error: Can&#39;t start duplicate Fluentd instance with the default config. To start Fluentd, please do one of the following: (Caution: Please be careful not to start multiple instances with the same config.) - Stop the Fluentd Windows service &#39;fluentdwinsvc&#39;. - Specify the config path explicitly by &#39;-c&#39; (&#39;--config&#39;). </code></pre> <p>Even though if you wrongly launch Fluentd to check version with <code>-v</code> (It should be <code>--version</code> to show version), then it causes the following error.</p> <pre><code>&gt; fluentd -v Error: Can&#39;t start duplicate Fluentd instance with the default config. To take the version, please use &#39;--version&#39;, not &#39;-v&#39; (&#39;--verbose&#39;). To start Fluentd, please do one of the following: (Caution: Please be careful not to start multiple instances with the same config.) - Stop the Fluentd Windows service &#39;fluentdwinsvc&#39;. - Specify the config path explicitly by &#39;-c&#39; (&#39;--config&#39;) </code></pre> <p>Note that this foolproof feature is intended to block launching duplicated Fluentd instance, you can explicitly launch Fluentd by specifying a specific option to pass it even though already Fluentd is running as a service.</p> <p>On Windows:</p> <ul> <li><code>-c</code> (<code>--config</code>)</li> <li><code>--dry-run</code></li> <li><code>--reg-winsvc</code></li> <li><code>--reg-winsvc-fluentdopt</code></li> <li><code>--show-plugin-config</code></li> </ul> <p>On Linux:</p> <ul> <li><code>-c</code> (<code>--config</code>)</li> <li><code>--dry-run</code></li> <li><code>--show-plugin-config</code></li> </ul> <h3>About next LTS schedule</h3> <p>We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.</p> <h3>About td-agent v4.5.2 and v4.5.3 (Windows)</h3> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>There is a exceptional maintenance release for v4.5.3 on Windows because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> 2024-07-02 <p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.4">v5.0.4</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.4 and a foolproof mechanism was implemented to prevent launching duplicated Fluentd instances, we recommend upgrading to fluent-package v5.0.4!</p> <h3>Changes from fluent-package v5.0.3</h3> <ul> <li>Update ruby to 3.2.4 (<a href="https://github.com/fluent/fluent-package-builder/pull/645">#645</a>)</li> <li>Fixed to prevent launching Fluentd wrongly if the service is already running (<a href="https://github.com/fluent/fluent-package-builder/pull/648">#648</a>,<a href="https://github.com/fluent/fluent-package-builder/pull/649">#649</a>)</li> <li>Added support for Ubuntu 24.04 (Noble Numbat)</li> </ul> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b> In this release, dropped CentOS 7 package because of EOL. </div> <h3>Fixed to prevent launching Fluentd wrongly if the service is already running</h3> <p>In this release, a foolproof mechanism was implemented to prevent launching Fluentd wrongly if the service is already running.</p> <p>As you know, you can check the version of Fluentd with <code>fluentd --version</code>, but there is a case that <code>fluentd -v</code> is executed wrongly to do it.</p> <p>When already running Fluentd as a service, <code>fluentd -v</code> launches a duplicated Fluentd instance with the same fluentd configuration. If you launch duplicated Fluentd instances, it causes the corruption of processing Fluentd buffer. To prevent such a situation, a foolproof was implemented now.</p> <p>For example, if Fluentd is running as a service, launching Fluentd causes an error to block it.</p> <p>Here is the example on Windows:</p> <pre><code>&gt; fluentd Error: Can&#39;t start duplicate Fluentd instance with the default config. To start Fluentd, please do one of the following: (Caution: Please be careful not to start multiple instances with the same config.) - Stop the Fluentd Windows service &#39;fluentdwinsvc&#39;. - Specify the config path explicitly by &#39;-c&#39; (&#39;--config&#39;). </code></pre> <p>Even though if you wrongly launch Fluentd to check version with <code>-v</code> (It should be <code>--version</code> to show version), then it causes the following error.</p> <pre><code>&gt; fluentd -v Error: Can&#39;t start duplicate Fluentd instance with the default config. To take the version, please use &#39;--version&#39;, not &#39;-v&#39; (&#39;--verbose&#39;). To start Fluentd, please do one of the following: (Caution: Please be careful not to start multiple instances with the same config.) - Stop the Fluentd Windows service &#39;fluentdwinsvc&#39;. - Specify the config path explicitly by &#39;-c&#39; (&#39;--config&#39;) </code></pre> <p>Note that this foolproof feature is intended to block launching duplicated Fluentd instance, you can explicitly launch Fluentd by specifying a specific option to pass it even though already Fluentd is running as a service.</p> <p>On Windows:</p> <ul> <li><code>-c</code> (<code>--config</code>)</li> <li><code>--dry-run</code></li> <li><code>--reg-winsvc</code></li> <li><code>--reg-winsvc-fluentdopt</code></li> <li><code>--show-plugin-config</code></li> </ul> <p>On Linux:</p> <ul> <li><code>-c</code> (<code>--config</code>)</li> <li><code>--dry-run</code></li> <li><code>--show-plugin-config</code></li> </ul> <h3>About next LTS schedule</h3> <p>We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.</p> <h3>About td-agent v4.5.2 and v4.5.3 (Windows)</h3> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>There is a exceptional maintenance release for v4.5.3 on Windows because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> Fluentd v1.17.0 has been released https://www.fluentd.org/blog/fluentd-v1.17.0-has-been-released <p>Hi users!</p> <p>We have released v1.17.0 on 2024-04-30. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430">here</a>.</p> <p>This release is a new release of v1.17 series. In this release, we added some new features for some plugins and fixed bugs of Parser.</p> <h2>Enhancement</h2> <h3><code>in_tail</code>: Add <code>glob_policy</code> option for expanding glob capability of <code>path</code> and <code>exclude_path</code></h3> <p>In this release, we added a new option <a href="https://docs.fluentd.org/input/tail#glob_policy">glob_policy</a> for <a href="https://docs.fluentd.org/input/tail">in_tail</a> plugin.</p> <p>In previous versions, we can use only <code>*</code> in glob patterns for <a href="https://docs.fluentd.org/input/tail#path">path</a> and <a href="https://docs.fluentd.org/input/tail#exclude_path">exclude_path</a> option.</p> <p>Example:</p> <pre><code>path /path/to/* exclude_path [&quot;/path/to/*.gz&quot;, &quot;/path/to/*.zip&quot;] </code></pre> <p>From this version, we can also use <code>[]</code>, <code>?</code>, and <code>{}</code> in glob patterns depending on the <code>glob_policy</code> option.</p> <p>Example:</p> <pre><code>path &quot;[0-1][2-3].log&quot; glob_policy extended </code></pre> <p>Please see <a href="https://docs.fluentd.org/input/tail#glob_policy">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4401">#4401</a> for more information.</p> <h3><code>out_http</code>: Support AWS Signature Version 4 authentication</h3> <p>In this release, we added a new option <code>aws_sigv4</code> for the <a href="https://docs.fluentd.org/output/http#method">method</a> setting of <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>By using this option, <code>out_http</code> can use <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html">AWS Signature Version 4</a>.</p> <p>For example, this allows <code>out_http</code> to write to <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html">Amazon OpenSearch Ingestion</a>.</p> <p>Please see <a href="https://docs.fluentd.org/output/http#method">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4459">#4459</a> for more information.</p> <h3><code>out_http</code>: Add option to reuse connections</h3> <p>In this release, we add a new option <a href="https://docs.fluentd.org/output/http#reuse_connections">reuse_connections</a> for <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>This option will improve throughput of <code>out_http</code>.</p> <p>Please see <a href="https://docs.fluentd.org/output/http#reuse_connections">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4330">#4330</a> for more information.</p> <h3><code>in_http</code>: Recognize CSP reports as JSON data</h3> <p>In this release, we make the data type of the request where the <code>Content-Type</code> is <code>application/csp-report</code> be considered JSON by default.</p> <p>Now, <code>in_http</code> can receive <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP">Content Security Policy</a>&#39;s report by default.</p> <p>Please see <a href="https://github.com/fluent/fluentd/pull/4282">#4282</a> for more information.</p> <h2>Bug Fixes</h2> <h3>Make sure parser returns hash</h3> <p>The record data in an event of Fluentd must be a hash object.</p> <ul> <li><a href="https://docs.fluentd.org/quickstart/life-of-a-fluentd-event">Life of a Fluentd event</a></li> </ul> <p>However, in the previous versions, some parser plugins could return a non-hash object, such as an array. It could cause errors in subsequent processing.</p> <p>In this release, the following parser plugins have been fixed.</p> <ul> <li><a href="https://docs.fluentd.org/parser/json">parser_json</a></li> <li><a href="https://docs.fluentd.org/parser/msgpack">parser_msgpack</a></li> </ul> <p>The changes are as follows:</p> <ul> <li>Make sure to return a hash record.</li> <li>Make sure to accept only a hash or an array of hash.</li> </ul> <p>Here are the details for each case.</p> <h4>Example of changed behavior</h4> <p>Config:</p> <pre><code>&lt;source&gt; @type tcp tag test.tcp &lt;parse&gt; @type json &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>Send an array data:</p> <pre><code>$ netcat 0.0.0.0 5170 [{&quot;k&quot;:&quot;v&quot;}, {&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result before this version:</p> <pre><code>{datetime} test.tcp: [{&quot;k&quot;:&quot;v&quot;},{&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result after this version:</p> <pre><code>{datetime} test.tcp: {&quot;k&quot;:&quot;v&quot;} {datetime} test.tcp: {&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h4>Example of resolved error</h4> <p>Config:</p> <pre><code>&lt;source&gt; @type tcp tag test.tcp &lt;parse&gt; @type json null_empty_string &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>Send an array data:</p> <pre><code>$ netcat 0.0.0.0 5170 [{&quot;k&quot;:&quot;v&quot;}, {&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result before this version:</p> <pre><code>{datetime} [error]: #0 unexpected error on reading data host=&quot;xxx&quot; port=xxx error_class=NoMethodError error=&quot;undefined method `each_key&#39; for [{\&quot;k\&quot;=&gt;\&quot;v\&quot;}, {\&quot;k2\&quot;=&gt;\&quot;v2\&quot;}]:Array&quot; </code></pre> <p>The result after this version:</p> <pre><code>{datetime} test.tcp: {&quot;k&quot;:&quot;v&quot;} {datetime} test.tcp: {&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h4>Remaining problem: filter_parser</h4> <p>In the previous versions, <code>filter_parser</code> could return an array record based on this wrong behavior. From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case. This behavior should improve in the future.</p> <p>Here is an example.</p> <pre><code>&lt;source&gt; @type sample tag test.array sample {&quot;message&quot;: &quot;[{\&quot;k\&quot;:\&quot;v\&quot;}, {\&quot;k2\&quot;:\&quot;v2\&quot;}]&quot;} &lt;/source&gt; &lt;filter test.**&gt; @type parser key_name message &lt;parse&gt; @type json &lt;/parse&gt; &lt;/filter&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>The result before this version:</p> <pre><code>{datetime} test.array: [{&quot;k&quot;:&quot;v&quot;},{&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result after this version:</p> <pre><code>{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=&quot;Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: &#39;[{\&quot;k\&quot;:\&quot;v\&quot;}, {\&quot;k2\&quot;:\&quot;v2\&quot;}]&#39;&quot; location=nil tag=&quot;test.array&quot; time=xxx record={&quot;k2&quot;=&gt;&quot;v2&quot;} {datetime} test.array: {&quot;k&quot;:&quot;v&quot;} </code></pre> <p>These are the major changes for this release.</p> <p>In addition, some performance improvements have been included! Please see <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430">ChangeLog</a> for details!</p> <p>Enjoy logging!</p> 2024-04-30 <p>Hi users!</p> <p>We have released v1.17.0 on 2024-04-30. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430">here</a>.</p> <p>This release is a new release of v1.17 series. In this release, we added some new features for some plugins and fixed bugs of Parser.</p> <h2>Enhancement</h2> <h3><code>in_tail</code>: Add <code>glob_policy</code> option for expanding glob capability of <code>path</code> and <code>exclude_path</code></h3> <p>In this release, we added a new option <a href="https://docs.fluentd.org/input/tail#glob_policy">glob_policy</a> for <a href="https://docs.fluentd.org/input/tail">in_tail</a> plugin.</p> <p>In previous versions, we can use only <code>*</code> in glob patterns for <a href="https://docs.fluentd.org/input/tail#path">path</a> and <a href="https://docs.fluentd.org/input/tail#exclude_path">exclude_path</a> option.</p> <p>Example:</p> <pre><code>path /path/to/* exclude_path [&quot;/path/to/*.gz&quot;, &quot;/path/to/*.zip&quot;] </code></pre> <p>From this version, we can also use <code>[]</code>, <code>?</code>, and <code>{}</code> in glob patterns depending on the <code>glob_policy</code> option.</p> <p>Example:</p> <pre><code>path &quot;[0-1][2-3].log&quot; glob_policy extended </code></pre> <p>Please see <a href="https://docs.fluentd.org/input/tail#glob_policy">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4401">#4401</a> for more information.</p> <h3><code>out_http</code>: Support AWS Signature Version 4 authentication</h3> <p>In this release, we added a new option <code>aws_sigv4</code> for the <a href="https://docs.fluentd.org/output/http#method">method</a> setting of <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>By using this option, <code>out_http</code> can use <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html">AWS Signature Version 4</a>.</p> <p>For example, this allows <code>out_http</code> to write to <a href="https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html">Amazon OpenSearch Ingestion</a>.</p> <p>Please see <a href="https://docs.fluentd.org/output/http#method">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4459">#4459</a> for more information.</p> <h3><code>out_http</code>: Add option to reuse connections</h3> <p>In this release, we add a new option <a href="https://docs.fluentd.org/output/http#reuse_connections">reuse_connections</a> for <a href="https://docs.fluentd.org/output/http">out_http</a> plugin.</p> <p>This option will improve throughput of <code>out_http</code>.</p> <p>Please see <a href="https://docs.fluentd.org/output/http#reuse_connections">the document</a> and <a href="https://github.com/fluent/fluentd/pull/4330">#4330</a> for more information.</p> <h3><code>in_http</code>: Recognize CSP reports as JSON data</h3> <p>In this release, we make the data type of the request where the <code>Content-Type</code> is <code>application/csp-report</code> be considered JSON by default.</p> <p>Now, <code>in_http</code> can receive <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP">Content Security Policy</a>&#39;s report by default.</p> <p>Please see <a href="https://github.com/fluent/fluentd/pull/4282">#4282</a> for more information.</p> <h2>Bug Fixes</h2> <h3>Make sure parser returns hash</h3> <p>The record data in an event of Fluentd must be a hash object.</p> <ul> <li><a href="https://docs.fluentd.org/quickstart/life-of-a-fluentd-event">Life of a Fluentd event</a></li> </ul> <p>However, in the previous versions, some parser plugins could return a non-hash object, such as an array. It could cause errors in subsequent processing.</p> <p>In this release, the following parser plugins have been fixed.</p> <ul> <li><a href="https://docs.fluentd.org/parser/json">parser_json</a></li> <li><a href="https://docs.fluentd.org/parser/msgpack">parser_msgpack</a></li> </ul> <p>The changes are as follows:</p> <ul> <li>Make sure to return a hash record.</li> <li>Make sure to accept only a hash or an array of hash.</li> </ul> <p>Here are the details for each case.</p> <h4>Example of changed behavior</h4> <p>Config:</p> <pre><code>&lt;source&gt; @type tcp tag test.tcp &lt;parse&gt; @type json &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>Send an array data:</p> <pre><code>$ netcat 0.0.0.0 5170 [{&quot;k&quot;:&quot;v&quot;}, {&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result before this version:</p> <pre><code>{datetime} test.tcp: [{&quot;k&quot;:&quot;v&quot;},{&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result after this version:</p> <pre><code>{datetime} test.tcp: {&quot;k&quot;:&quot;v&quot;} {datetime} test.tcp: {&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h4>Example of resolved error</h4> <p>Config:</p> <pre><code>&lt;source&gt; @type tcp tag test.tcp &lt;parse&gt; @type json null_empty_string &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>Send an array data:</p> <pre><code>$ netcat 0.0.0.0 5170 [{&quot;k&quot;:&quot;v&quot;}, {&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result before this version:</p> <pre><code>{datetime} [error]: #0 unexpected error on reading data host=&quot;xxx&quot; port=xxx error_class=NoMethodError error=&quot;undefined method `each_key&#39; for [{\&quot;k\&quot;=&gt;\&quot;v\&quot;}, {\&quot;k2\&quot;=&gt;\&quot;v2\&quot;}]:Array&quot; </code></pre> <p>The result after this version:</p> <pre><code>{datetime} test.tcp: {&quot;k&quot;:&quot;v&quot;} {datetime} test.tcp: {&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h4>Remaining problem: filter_parser</h4> <p>In the previous versions, <code>filter_parser</code> could return an array record based on this wrong behavior. From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case. This behavior should improve in the future.</p> <p>Here is an example.</p> <pre><code>&lt;source&gt; @type sample tag test.array sample {&quot;message&quot;: &quot;[{\&quot;k\&quot;:\&quot;v\&quot;}, {\&quot;k2\&quot;:\&quot;v2\&quot;}]&quot;} &lt;/source&gt; &lt;filter test.**&gt; @type parser key_name message &lt;parse&gt; @type json &lt;/parse&gt; &lt;/filter&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>The result before this version:</p> <pre><code>{datetime} test.array: [{&quot;k&quot;:&quot;v&quot;},{&quot;k2&quot;:&quot;v2&quot;}] </code></pre> <p>The result after this version:</p> <pre><code>{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=&quot;Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: &#39;[{\&quot;k\&quot;:\&quot;v\&quot;}, {\&quot;k2\&quot;:\&quot;v2\&quot;}]&#39;&quot; location=nil tag=&quot;test.array&quot; time=xxx record={&quot;k2&quot;=&gt;&quot;v2&quot;} {datetime} test.array: {&quot;k&quot;:&quot;v&quot;} </code></pre> <p>These are the major changes for this release.</p> <p>In addition, some performance improvements have been included! Please see <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430">ChangeLog</a> for details!</p> <p>Enjoy logging!</p> fluent-package v5.0.3 has been released https://www.fluentd.org/blog/fluent-package-v5.0.3-has-been-released <p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.3">v5.0.3</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As significant slow starting service and crash issues during startup on Windows were fixed, we recommend upgrading to fluent-package v5.0.3!</p> <h3>Changes from fluent-package v5.0.2</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.5">1.16.5</a>. See the following blog articles about details. <ul> <li><a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a></li> <li><a href="fluentd-v1.16.5-have-been-released">Fluentd v1.16.5 has been released</a></li> </ul></li> <li>Update bundled plugins <ul> <li>e.g. fluent-diagtool v1.0.5. It supports to collect list of plugins on Windows.</li> </ul></li> <li>msi: fixed wrong environment path for Fluent Package Prompt (<a href="https://github.com/fluent/fluent-package-builder/pull/606">#606</a>) <ul> <li>It breaks fluent-diagtool behavior to launch fluent-gem correctly.</li> </ul></li> <li>msi: removed unnecessary path delimiter (<a href="https://github.com/fluent/fluent-package-builder/pull/607">#607</a>) <ul> <li>It doesn&#39;t cause any problem yet, but it should treat <code>%~dp0</code> correctly.</li> </ul></li> <li>rpm: fixed to take over enabled state of systemd service from td-agent v4 (<a href="https://github.com/fluent/fluent-package-builder/pull/613">#613</a>)</li> <li>deb rpm: fixed to quote target files correctly not to cause migration failures (<a href="https://github.com/fluent/fluent-package-builder/pull/615">#615</a>)</li> <li>msi: added a patch for RubyInstaller to avoid crash on start up (<a href="https://github.com/fluent/fluent-package-builder/pull/620">#620</a>)</li> <li>msi: fixed slow start issue on Windows (<a href="https://github.com/fluent/fluent-package-builder/pull/631">#631</a>)</li> </ul> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b> v5.0.3 for Windows msi will be shipped later. </div> <h3>About next LTS schedule</h3> <p>We plan to ship next LTS version of fluent-package v5.0.4 on June, 2024. The content of updates are still in T.B.D.</p> <h3>About td-agent v4.5.2 and v4.5.3 (Windows)</h3> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>There is a exceptional maintenance release for v4.5.3 on Windows because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> 2024-03-29 <p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.3">v5.0.3</a>. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)</p> <p>This is a maintenance release of v5.0.x LTS series. As significant slow starting service and crash issues during startup on Windows were fixed, we recommend upgrading to fluent-package v5.0.3!</p> <h3>Changes from fluent-package v5.0.2</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.5">1.16.5</a>. See the following blog articles about details. <ul> <li><a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a></li> <li><a href="fluentd-v1.16.5-have-been-released">Fluentd v1.16.5 has been released</a></li> </ul></li> <li>Update bundled plugins <ul> <li>e.g. fluent-diagtool v1.0.5. It supports to collect list of plugins on Windows.</li> </ul></li> <li>msi: fixed wrong environment path for Fluent Package Prompt (<a href="https://github.com/fluent/fluent-package-builder/pull/606">#606</a>) <ul> <li>It breaks fluent-diagtool behavior to launch fluent-gem correctly.</li> </ul></li> <li>msi: removed unnecessary path delimiter (<a href="https://github.com/fluent/fluent-package-builder/pull/607">#607</a>) <ul> <li>It doesn&#39;t cause any problem yet, but it should treat <code>%~dp0</code> correctly.</li> </ul></li> <li>rpm: fixed to take over enabled state of systemd service from td-agent v4 (<a href="https://github.com/fluent/fluent-package-builder/pull/613">#613</a>)</li> <li>deb rpm: fixed to quote target files correctly not to cause migration failures (<a href="https://github.com/fluent/fluent-package-builder/pull/615">#615</a>)</li> <li>msi: added a patch for RubyInstaller to avoid crash on start up (<a href="https://github.com/fluent/fluent-package-builder/pull/620">#620</a>)</li> <li>msi: fixed slow start issue on Windows (<a href="https://github.com/fluent/fluent-package-builder/pull/631">#631</a>)</li> </ul> <div markdown="span" class="alert alert-info" role="alert"> <i class="fa fa-info-circle"></i> <b>Note:</b> v5.0.3 for Windows msi will be shipped later. </div> <h3>About next LTS schedule</h3> <p>We plan to ship next LTS version of fluent-package v5.0.4 on June, 2024. The content of updates are still in T.B.D.</p> <h3>About td-agent v4.5.2 and v4.5.3 (Windows)</h3> <p>As it was already announced <a href="schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>, td-agent v4 was reached EOL in Dec, 2023.</p> <p>There is a exceptional maintenance release for v4.5.3 on Windows because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case.</p> <p>We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See <a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> Fluentd v1.16.5 has been released https://www.fluentd.org/blog/fluentd-v1.16.5-have-been-released <p>Hi users!</p> <p>We have released v1.16.5 on 2024-03-27. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1165---20240327">here</a>.</p> <p>The previous version v1.16.4 has a serious Buffer bug when it processes large data exceeding chunk size limit. So, we have released v1.16.5 urgently, and have fixed the bug. So, <strong>please don&#39;t use v1.16.4</strong>.</p> <p>Sorry for troubling.</p> <h2>Bug Fixes</h2> <h3>Buffer: Fix emit error of v1.16.4 sometimes failing to process large data exceeding chunk size limit</h3> <p>In the previous version v1.16.4, we fixed a Buffer bug (Please see <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a> for details).</p> <p>There was a race condition problem with that fix, and similar Buffer errors explained in <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a> could occur.</p> <ul> <li><code>emit transaction failed: ...</code></li> <li><code>send an error event stream to @ERROR: ...</code></li> </ul> <p>These errors are the same as the Buffer error of <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a>, but it would be more likely to happen. The cause is a race condition when processing large data exceeding chunk size limit. So, in that case, these errors can occur depending on the timing.</p> <p>So, please don&#39;t use v1.16.4.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4447">#4447</a> for more information.</p> <p>Sorry for troubling.</p> 2024-03-27 <p>Hi users!</p> <p>We have released v1.16.5 on 2024-03-27. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1165---20240327">here</a>.</p> <p>The previous version v1.16.4 has a serious Buffer bug when it processes large data exceeding chunk size limit. So, we have released v1.16.5 urgently, and have fixed the bug. So, <strong>please don&#39;t use v1.16.4</strong>.</p> <p>Sorry for troubling.</p> <h2>Bug Fixes</h2> <h3>Buffer: Fix emit error of v1.16.4 sometimes failing to process large data exceeding chunk size limit</h3> <p>In the previous version v1.16.4, we fixed a Buffer bug (Please see <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a> for details).</p> <p>There was a race condition problem with that fix, and similar Buffer errors explained in <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a> could occur.</p> <ul> <li><code>emit transaction failed: ...</code></li> <li><code>send an error event stream to @ERROR: ...</code></li> </ul> <p>These errors are the same as the Buffer error of <a href="fluentd-v1.16.4-have-been-released">Fluentd v1.16.4 has been released</a>, but it would be more likely to happen. The cause is a race condition when processing large data exceeding chunk size limit. So, in that case, these errors can occur depending on the timing.</p> <p>So, please don&#39;t use v1.16.4.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4447">#4447</a> for more information.</p> <p>Sorry for troubling.</p> Fluentd v1.16.4 has been released https://www.fluentd.org/blog/fluentd-v1.16.4-have-been-released <p>Hi users!</p> <p>We have released v1.16.4 on 2024-03-14. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1164---20240314">here</a>.</p> <p>In this release, we fixed several bugs and improved performance.</p> <h2>Bug Fixes</h2> <h3>Buffer: Fix emit error sometimes caused by failing to process large data exceeding chunk size limit</h3> <p>In previous versions, when Buffer processes large data exceeding the chunk size limit, it sometimes fails and causes an emit error.</p> <p>It is usually unlikely, but it may occur when Buffer receives data with extremely large records that are distributed unevenly in the data.</p> <p>If you set <a href="https://docs.fluentd.org/configuration/config-file#error-label">@ERROR</a> label, the data is routed to that label, and Fluentd outputs a warning log message: <code>send an error event stream to @ERROR: ...</code>.</p> <p>However, if you don&#39;t set <code>@ERROR</code> label, <strong>the data is discarded</strong>, and Fluentd outputs a warning log message: <code>emit transaction failed: ...</code>. In addition, Input plugins that do not consider an emit error <strong>may stop working</strong>, such as <code>in_windows_eventlog2</code>.</p> <p>Here is an example of the warning message:</p> <pre><code>2024-03-22 14:13:35 +0900 [warn]: #0 emit transaction failed: error_class=IOError error=&quot;closed stream&quot; location=&quot;/path/to/fluentd/lib/fluent/plugin/buffer/file_chunk.rb:82:in `pos&#39;&quot; tag=&quot;test&quot; </code></pre> <p>If you see these warnings, please update Fluentd.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4342">#4342</a> for more information.</p> <h3>Others</h3> <ul> <li><code>in_tail</code>: Fix tail watchers in <code>rotate_wait</code> state not being managed. <a href="https://github.com/fluent/fluentd/pull/4334">#4334</a> <ul> <li>This problem should not have any actual negative impact on the operation.</li> </ul></li> <li>Buffer: Avoid unnecessary log processing. It will improve performance. <a href="https://github.com/fluent/fluentd/pull/4331">#4331</a></li> </ul> <p>Enjoy logging!</p> 2024-03-14 <p>Hi users!</p> <p>We have released v1.16.4 on 2024-03-14. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1164---20240314">here</a>.</p> <p>In this release, we fixed several bugs and improved performance.</p> <h2>Bug Fixes</h2> <h3>Buffer: Fix emit error sometimes caused by failing to process large data exceeding chunk size limit</h3> <p>In previous versions, when Buffer processes large data exceeding the chunk size limit, it sometimes fails and causes an emit error.</p> <p>It is usually unlikely, but it may occur when Buffer receives data with extremely large records that are distributed unevenly in the data.</p> <p>If you set <a href="https://docs.fluentd.org/configuration/config-file#error-label">@ERROR</a> label, the data is routed to that label, and Fluentd outputs a warning log message: <code>send an error event stream to @ERROR: ...</code>.</p> <p>However, if you don&#39;t set <code>@ERROR</code> label, <strong>the data is discarded</strong>, and Fluentd outputs a warning log message: <code>emit transaction failed: ...</code>. In addition, Input plugins that do not consider an emit error <strong>may stop working</strong>, such as <code>in_windows_eventlog2</code>.</p> <p>Here is an example of the warning message:</p> <pre><code>2024-03-22 14:13:35 +0900 [warn]: #0 emit transaction failed: error_class=IOError error=&quot;closed stream&quot; location=&quot;/path/to/fluentd/lib/fluent/plugin/buffer/file_chunk.rb:82:in `pos&#39;&quot; tag=&quot;test&quot; </code></pre> <p>If you see these warnings, please update Fluentd.</p> <p>See <a href="https://github.com/fluent/fluentd/pull/4342">#4342</a> for more information.</p> <h3>Others</h3> <ul> <li><code>in_tail</code>: Fix tail watchers in <code>rotate_wait</code> state not being managed. <a href="https://github.com/fluent/fluentd/pull/4334">#4334</a> <ul> <li>This problem should not have any actual negative impact on the operation.</li> </ul></li> <li>Buffer: Avoid unnecessary log processing. It will improve performance. <a href="https://github.com/fluent/fluentd/pull/4331">#4331</a></li> </ul> <p>Enjoy logging!</p> Open Source Summit Japan 2023 https://www.fluentd.org/blog/ossjapan-2023 <p>Hi users and developers!</p> <p>We would like to share the information about an event appearance of Fluentd maintainers.</p> <p>Daijiro Fukuda and Kentaro Hayashi, Fluentd Maintainers from ClearCode Inc., are going to talk at the Open Source Summit Japan on December 5th</p> <p>The event has been taken thousands of attendees all over the world and one of the leading Open Source event in Japan. We are honored to take this opportunity and very excited to share about Fluentd project at the conference!</p> <p>Everyone attending the event, please join us!</p> <h3>About the Talk</h3> <p><strong>Date</strong>: December 5th 14:00-14:40<br> <strong>Speakers</strong>: Daijiro Fukuda and Kentaro Hayashi from <a href="https://www.clear-code.com/">ClearCode Inc.</a><br> <strong>Summary</strong>:<br> Fluentd, the CNCF graduated project, starts a new chapter as the OSS project. Fluentd community released the new package fluent-package as the successor of td-agent and started providing a new release cycle to fit better with needs. This talk is going to share the latest news of Fluentd, including its updates, package, rebranding and so on, with people who are using Fluentd or struggling to manage their logs. Fluentd is an open source data collector that unifies data collection and consumption. Because of its flexible plugin system and scalability, Fluentd is adopted in various systems and has become a de facto standard, especially for scale logging in distributed systems.</p> <p>In the new release cycle, the package applies only bug fixes and security fixes of Fluentd over a long term, about two years. We are confident that this will meet the demands of users who want to use Fluentd stably over the long term. In this session, we will talk about (a) the history of Fluentd v1 and td-agent v4, (b) why Fluentd community decided to rebrand existing td-agent to fluent-package, (c) what the benefits and impacts of the new release cycle are, and (d) how to upgrade to fluent-package from td-agent.</p> <p>Detail info: <a href="https://ossjapan2023.sched.com/event/1Typb/new-chapter-of-fluentd-rebranding-and-new-release-cycle-lts-daijiro-fukuda-kentaro-hayashi-clearcode-inc?iframe=no&w=100%&sidebar=yes&bg=no">Link to event page</a></p> <h3>Overall Event Information</h3> <p><strong>Name</strong>: Open Source Summit Japan 2023<br> <strong>Date</strong>: 2023-12-04 to 2023-12-06<br> <strong>Host</strong>: The Linux Foundation<br> <strong>Description</strong>:</p> <blockquote> <p>Open Source Summit Japan is the leading conference in Japan connecting the open source ecosystem under one roof, providing a forum for technologists and open source industry leaders to collaborate and share information, learn about the latest in open source technologies and find out how to gain a competitive advantage by using innovative open solutions.</p> <p>Open Source Summit is a conference umbrella, composed of a collection of events covering the most important technologies, topics, and issues affecting open source today.</p> </blockquote> <p>Quote from the event web page : <a href="https://events.linuxfoundation.org/open-source-summit-japan/">https://events.linuxfoundation.org/open-source-summit-japan/</a></p> 2023-12-04 <p>Hi users and developers!</p> <p>We would like to share the information about an event appearance of Fluentd maintainers.</p> <p>Daijiro Fukuda and Kentaro Hayashi, Fluentd Maintainers from ClearCode Inc., are going to talk at the Open Source Summit Japan on December 5th</p> <p>The event has been taken thousands of attendees all over the world and one of the leading Open Source event in Japan. We are honored to take this opportunity and very excited to share about Fluentd project at the conference!</p> <p>Everyone attending the event, please join us!</p> <h3>About the Talk</h3> <p><strong>Date</strong>: December 5th 14:00-14:40<br> <strong>Speakers</strong>: Daijiro Fukuda and Kentaro Hayashi from <a href="https://www.clear-code.com/">ClearCode Inc.</a><br> <strong>Summary</strong>:<br> Fluentd, the CNCF graduated project, starts a new chapter as the OSS project. Fluentd community released the new package fluent-package as the successor of td-agent and started providing a new release cycle to fit better with needs. This talk is going to share the latest news of Fluentd, including its updates, package, rebranding and so on, with people who are using Fluentd or struggling to manage their logs. Fluentd is an open source data collector that unifies data collection and consumption. Because of its flexible plugin system and scalability, Fluentd is adopted in various systems and has become a de facto standard, especially for scale logging in distributed systems.</p> <p>In the new release cycle, the package applies only bug fixes and security fixes of Fluentd over a long term, about two years. We are confident that this will meet the demands of users who want to use Fluentd stably over the long term. In this session, we will talk about (a) the history of Fluentd v1 and td-agent v4, (b) why Fluentd community decided to rebrand existing td-agent to fluent-package, (c) what the benefits and impacts of the new release cycle are, and (d) how to upgrade to fluent-package from td-agent.</p> <p>Detail info: <a href="https://ossjapan2023.sched.com/event/1Typb/new-chapter-of-fluentd-rebranding-and-new-release-cycle-lts-daijiro-fukuda-kentaro-hayashi-clearcode-inc?iframe=no&w=100%&sidebar=yes&bg=no">Link to event page</a></p> <h3>Overall Event Information</h3> <p><strong>Name</strong>: Open Source Summit Japan 2023<br> <strong>Date</strong>: 2023-12-04 to 2023-12-06<br> <strong>Host</strong>: The Linux Foundation<br> <strong>Description</strong>:</p> <blockquote> <p>Open Source Summit Japan is the leading conference in Japan connecting the open source ecosystem under one roof, providing a forum for technologists and open source industry leaders to collaborate and share information, learn about the latest in open source technologies and find out how to gain a competitive advantage by using innovative open solutions.</p> <p>Open Source Summit is a conference umbrella, composed of a collection of events covering the most important technologies, topics, and issues affecting open source today.</p> </blockquote> <p>Quote from the event web page : <a href="https://events.linuxfoundation.org/open-source-summit-japan/">https://events.linuxfoundation.org/open-source-summit-japan/</a></p> fluent-package v5.0.2 has been released https://www.fluentd.org/blog/fluent-package-v5.0.2-has-been-released <p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.2">v5.0.2</a> and td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.5.2">v4.5.2</a>. fluent-package is a stable distribution package of Fluentd. </p> <p>This is a maintenance release of v5.0.x LTS series. As significant <code>in_tail</code> bugs (wrongly stopping tailing logs) were fixed in latest release, we recommend upgrading to fluent-package v5.0.2!</p> <h3>Changes from fluent-package v5.0.1</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.3">1.16.3</a> which contains significant bug fixes about <code>in_tail</code>. See <a href="/blog/fluentd-v1.16.2-v1.16.3-have-been-released">Fluentd v1.16.3 and v1.16.2 have been released</a> blog article about details.</li> <li>Update plugins <ul> <li>fluent-diagtool v1.0.3. It supports fluent-package and can collect information about locally installed gems. It may help to migrate from td-agent v4 a bit. See <a href="/blog/upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a> for migration.</li> </ul></li> <li>msi: support path which contains space or parenthesis (<a href="https://github.com/fluent/fluent-package-builder/pull/589">#589</a>)</li> <li>deb: fixed system user/group name in logrotate config (<a href="https://github.com/fluent/fluent-package-builder/pull/592">#592</a>,<a href="https://github.com/fluent/fluent-package-builder/pull/594">#594</a>) <ul> <li>It fixes a bug that unknown user error was reported during log rotation.</li> </ul></li> <li>rpm: fixed to create fluentd user as system account (<a href="https://github.com/fluent/fluent-package-builder/pull/596">#596</a>) <ul> <li>It fixes a bug that /var/lib/fluent directory was created unexpectedly. It doesn&#39;t affect the fluentd service behavior, but it is desirable one.</li> </ul></li> <li>rpm: changed to keep system account after removing fluent-package. (<a href="https://github.com/fluent/fluent-package-builder/pull/598">#598</a>) <ul> <li>In the previous versions, there was a bug that group was not cleanly removed when the package was upgraded from td-agent v4. This change makes reinstall/downgrade friendly.</li> </ul></li> </ul> <h3>About next LTS schedule</h3> <p>We plan to ship next LTS version of fluent-package v5.0.3 on Feb, 2024. The content of updates are still in T.B.D.</p> <h3>About td-agent v4.5.2</h3> <p>This is a exceptional maintenance release of v4.5.x series. Fluentd was updated to 1.16.3 because it contains significant bug fixes about <code>in_tail</code>. Note that td-agent will not be updated anymore. See <a href="https://www.fluentd.org/blog/schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>.</p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> 2023-11-29 <p>Hi users!</p> <p>We have released fluent-package <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.2">v5.0.2</a> and td-agent <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v4.5.2">v4.5.2</a>. fluent-package is a stable distribution package of Fluentd. </p> <p>This is a maintenance release of v5.0.x LTS series. As significant <code>in_tail</code> bugs (wrongly stopping tailing logs) were fixed in latest release, we recommend upgrading to fluent-package v5.0.2!</p> <h3>Changes from fluent-package v5.0.1</h3> <ul> <li>Update fluentd to <a href="https://github.com/fluent/fluentd/releases/tag/v1.16.3">1.16.3</a> which contains significant bug fixes about <code>in_tail</code>. See <a href="/blog/fluentd-v1.16.2-v1.16.3-have-been-released">Fluentd v1.16.3 and v1.16.2 have been released</a> blog article about details.</li> <li>Update plugins <ul> <li>fluent-diagtool v1.0.3. It supports fluent-package and can collect information about locally installed gems. It may help to migrate from td-agent v4 a bit. See <a href="/blog/upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a> for migration.</li> </ul></li> <li>msi: support path which contains space or parenthesis (<a href="https://github.com/fluent/fluent-package-builder/pull/589">#589</a>)</li> <li>deb: fixed system user/group name in logrotate config (<a href="https://github.com/fluent/fluent-package-builder/pull/592">#592</a>,<a href="https://github.com/fluent/fluent-package-builder/pull/594">#594</a>) <ul> <li>It fixes a bug that unknown user error was reported during log rotation.</li> </ul></li> <li>rpm: fixed to create fluentd user as system account (<a href="https://github.com/fluent/fluent-package-builder/pull/596">#596</a>) <ul> <li>It fixes a bug that /var/lib/fluent directory was created unexpectedly. It doesn&#39;t affect the fluentd service behavior, but it is desirable one.</li> </ul></li> <li>rpm: changed to keep system account after removing fluent-package. (<a href="https://github.com/fluent/fluent-package-builder/pull/598">#598</a>) <ul> <li>In the previous versions, there was a bug that group was not cleanly removed when the package was upgraded from td-agent v4. This change makes reinstall/downgrade friendly.</li> </ul></li> </ul> <h3>About next LTS schedule</h3> <p>We plan to ship next LTS version of fluent-package v5.0.3 on Feb, 2024. The content of updates are still in T.B.D.</p> <h3>About td-agent v4.5.2</h3> <p>This is a exceptional maintenance release of v4.5.x series. Fluentd was updated to 1.16.3 because it contains significant bug fixes about <code>in_tail</code>. Note that td-agent will not be updated anymore. See <a href="https://www.fluentd.org/blog/schedule-for-td-agent-4-eol">Drop schedule announcement about EOL of Treasure Agent (td-agent) 4</a>.</p> <h3>Download</h3> <p>Please see <a href="/download/fluent_package">the download page</a>.</p> Fluentd v1.16.3 and v1.16.2 have been released https://www.fluentd.org/blog/fluentd-v1.16.2-v1.16.3-have-been-released <p>Hi users!</p> <p>We have released v1.16.3 (<a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1163---20231114">changelog</a>) on 2023-11-14. We also describe the content of v1.16.2 (<a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1162---20230714">changelog</a>) on 2023-07-14 here.</p> <p>In these releases, <code>in_tail</code> issues that have been reported for years are fixed! And other some bug fixes too.</p> <h2>Bug Fixes</h2> <h3><code>in_tail</code>: Fix <code>in_tail</code> wrongly stopping tailing the current target file and causing handle leaks</h3> <p>This issue has been reported for years. It was looking very serious, but the cause remained unclear until recently.</p> <p>At last, we found different causes when the <a href="https://docs.fluentd.org/input/tail#follow_inodes">follow_inodes option</a> is <code>true</code> and when it is <code>false</code> (default). We fixed the problem of <code>follow_inodes true</code> case in v1.16.2, and fixed the problem of <code>follow_inodes false</code> case in v1.16.3.</p> <p>Especially, in <code>follow_inodes false</code> case, the trouble was mainly caused by <a href="https://docs.fluentd.org/input/tail#rotate_wait">rotate_wait option</a>. In past versions, please avoid using large value for this option. In particular, if you make it greater than <a href="https://docs.fluentd.org/input/tail#refresh_interval">refresh_interval option</a>, it will surely cause this issue.</p> <p>In Fluentd v1.15.1 (td-agent v4.4.0) or later, the following warning log is recorded when this issue occurs.</p> <pre><code>Skip update_watcher because watcher has been already updated by other inotify event </code></pre> <p>If this issue occurs, please restart Fluentd, and please update Fluentd to the latest version. fluent-package v5.0.2 will be released in this month and it contains these fixes.</p> <p>See the following PRs for more information.</p> <ul> <li><code>follow_inodes true</code> case: <a href="https://github.com/fluent/fluentd/pull/4208">#4208</a></li> <li><code>follow_inodes false</code> case: <a href="https://github.com/fluent/fluentd/pull/4327">#4327</a></li> </ul> <h3>Others</h3> <ul> <li><code>in_tail</code>: Fix possible log duplication of <a href="https://docs.fluentd.org/input/tail#follow_inodes">follow_inodes option</a>. <a href="https://github.com/fluent/fluentd/pull/4237">#4237</a>, <a href="https://github.com/fluent/fluentd/pull/4239">#4239</a></li> <li><code>in_forward</code>: Fix corrupted data possibly breaking other data. <a href="https://github.com/fluent/fluentd/pull/4178">#4178</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#ignore_same_log_interval">ignore_same_log_interval option</a>: Fix cache size growing unlimitedly. <a href="https://github.com/fluent/fluentd/pull/4229">#4229</a></li> <li><a href="https://docs.fluentd.org/deployment/trouble-shooting#dump-fluentds-internal-information">SIGDUMP</a>: Fix wrong value of object counts. <a href="https://github.com/fluent/fluentd/pull/4225">#4225</a></li> <li>Windows: Fix Fluentd failing to start when the log path isn&#39;t specified in the command line. <a href="https://github.com/fluent/fluentd/pull/4188">#4188</a></li> <li>Buffer: Fix <code>NoMethodError</code>. <a href="https://github.com/fluent/fluentd/pull/4303">#4303</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#rotate_age">rotate_age option</a>: Fix enum values causing setting error. <a href="https://github.com/fluent/fluentd/pull/4311">#4311</a></li> </ul> <p>Enjoy logging!</p> 2023-11-14 <p>Hi users!</p> <p>We have released v1.16.3 (<a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1163---20231114">changelog</a>) on 2023-11-14. We also describe the content of v1.16.2 (<a href="https://github.com/fluent/fluentd/blob/v1.16/CHANGELOG.md#release-v1162---20230714">changelog</a>) on 2023-07-14 here.</p> <p>In these releases, <code>in_tail</code> issues that have been reported for years are fixed! And other some bug fixes too.</p> <h2>Bug Fixes</h2> <h3><code>in_tail</code>: Fix <code>in_tail</code> wrongly stopping tailing the current target file and causing handle leaks</h3> <p>This issue has been reported for years. It was looking very serious, but the cause remained unclear until recently.</p> <p>At last, we found different causes when the <a href="https://docs.fluentd.org/input/tail#follow_inodes">follow_inodes option</a> is <code>true</code> and when it is <code>false</code> (default). We fixed the problem of <code>follow_inodes true</code> case in v1.16.2, and fixed the problem of <code>follow_inodes false</code> case in v1.16.3.</p> <p>Especially, in <code>follow_inodes false</code> case, the trouble was mainly caused by <a href="https://docs.fluentd.org/input/tail#rotate_wait">rotate_wait option</a>. In past versions, please avoid using large value for this option. In particular, if you make it greater than <a href="https://docs.fluentd.org/input/tail#refresh_interval">refresh_interval option</a>, it will surely cause this issue.</p> <p>In Fluentd v1.15.1 (td-agent v4.4.0) or later, the following warning log is recorded when this issue occurs.</p> <pre><code>Skip update_watcher because watcher has been already updated by other inotify event </code></pre> <p>If this issue occurs, please restart Fluentd, and please update Fluentd to the latest version. fluent-package v5.0.2 will be released in this month and it contains these fixes.</p> <p>See the following PRs for more information.</p> <ul> <li><code>follow_inodes true</code> case: <a href="https://github.com/fluent/fluentd/pull/4208">#4208</a></li> <li><code>follow_inodes false</code> case: <a href="https://github.com/fluent/fluentd/pull/4327">#4327</a></li> </ul> <h3>Others</h3> <ul> <li><code>in_tail</code>: Fix possible log duplication of <a href="https://docs.fluentd.org/input/tail#follow_inodes">follow_inodes option</a>. <a href="https://github.com/fluent/fluentd/pull/4237">#4237</a>, <a href="https://github.com/fluent/fluentd/pull/4239">#4239</a></li> <li><code>in_forward</code>: Fix corrupted data possibly breaking other data. <a href="https://github.com/fluent/fluentd/pull/4178">#4178</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#ignore_same_log_interval">ignore_same_log_interval option</a>: Fix cache size growing unlimitedly. <a href="https://github.com/fluent/fluentd/pull/4229">#4229</a></li> <li><a href="https://docs.fluentd.org/deployment/trouble-shooting#dump-fluentds-internal-information">SIGDUMP</a>: Fix wrong value of object counts. <a href="https://github.com/fluent/fluentd/pull/4225">#4225</a></li> <li>Windows: Fix Fluentd failing to start when the log path isn&#39;t specified in the command line. <a href="https://github.com/fluent/fluentd/pull/4188">#4188</a></li> <li>Buffer: Fix <code>NoMethodError</code>. <a href="https://github.com/fluent/fluentd/pull/4303">#4303</a></li> <li><a href="https://docs.fluentd.org/deployment/system-config#rotate_age">rotate_age option</a>: Fix enum values causing setting error. <a href="https://github.com/fluent/fluentd/pull/4311">#4311</a></li> </ul> <p>Enjoy logging!</p> Drop schedule announcement about EOL of Treasure Agent (td-agent) 4 https://www.fluentd.org/blog/schedule-for-td-agent-4-eol <p>Hi users and developers!</p> <p>We announce the dropping schedule for TD Agent 4 development.</p> <h2>About Treasure Agent (td-agent) 4</h2> <p>Since td-agent 4.0.0 was released in 2020, td-agent 4 has been maintained for a long time.</p> <p>As new major version of td-agent 4 was rebranded (to <code>fluent-package</code> v5), and released in August 2023, we decided to stop maintaining td-agent 4.</p> <h2>Scheduled end of life - Dec 2023</h2> <p>As you know, already stepping down maintenance activity, new minor update for td-agent 4 will not be shipped anymore. (td-agent v4.5.1 will be the last release of this series, it will be shipped in this month.)</p> <p>Thus, we recommend using fluent-package v5 for new deployment :)</p> <h2>How to migrate to Fluent Package v5</h2> <p>There is a good article to do it.</p> <ul> <li><a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></li> </ul> <p>Follow the above instructions.</p> 2023-08-29 <p>Hi users and developers!</p> <p>We announce the dropping schedule for TD Agent 4 development.</p> <h2>About Treasure Agent (td-agent) 4</h2> <p>Since td-agent 4.0.0 was released in 2020, td-agent 4 has been maintained for a long time.</p> <p>As new major version of td-agent 4 was rebranded (to <code>fluent-package</code> v5), and released in August 2023, we decided to stop maintaining td-agent 4.</p> <h2>Scheduled end of life - Dec 2023</h2> <p>As you know, already stepping down maintenance activity, new minor update for td-agent 4 will not be shipped anymore. (td-agent v4.5.1 will be the last release of this series, it will be shipped in this month.)</p> <p>Thus, we recommend using fluent-package v5 for new deployment :)</p> <h2>How to migrate to Fluent Package v5</h2> <p>There is a good article to do it.</p> <ul> <li><a href="upgrade-td-agent-v4-to-v5">Upgrade to fluent-package v5</a></li> </ul> <p>Follow the above instructions.</p> Scheduled support lifecycle announcement about Fluent Package https://www.fluentd.org/blog/fluent-package-scheduled-lifecycle <p>Hi users!</p> <p>We have released <code>fluent-package</code> <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.0">v5.0.0</a> (Release Candidate) and the following <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.1">v5.0.1</a> (General Availability). fluent-package is formerly known as <code>td-agent</code> and a stable distribution package of Fluentd.</p> <p>In this blog article, we explains why the concept of channels is introduced.</p> <h3>Why is LTS (Long Term Support) channel introduced?</h3> <p>In the previous versions, <code>td-agent</code> was shipped as quarterly releases in most cases. It sometimes contains fluentd minor upgrade occasionally, which means that not only bug or security fixes, but also contains new features. As a result, there was a case that the stability was affected unexpectedly.</p> <p>In point view of adopting fluentd (and <code>td-agent</code>) for enterprise services, there was a requirement for the distribution which contains security or bug fixes only and scheduled lifecycle of it.</p> <p>The LTS release is the answer to support them.</p> <h3>Channels for updates</h3> <p>There are two channels of support lifecycle.</p> <ul> <li>Normal release</li> <li>LTS (Long Term Support)</li> </ul> <p>Here is the difference of these channels.</p> <ul> <li>Normal release (5.x series) <ul> <li>Recommended for early adopters (eager to try new features of Fluentd)</li> <li>Rapid release development (planned semi-annually releases, but it may vary.)</li> <li>Fluentd will be upgraded occasionally <ul> <li>Minor upgrade will be applied. e.g. 1.16 =&gt; 1.17 and so on.</li> <li>e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), ... v5.1.0 (Fluentd v1.17.0)</li> </ul></li> <li>The only latest version will be supported (bug fix will not be backported into older version)</li> </ul></li> <li>LTS (Long Term Support, 5.0.x series) <ul> <li>Recommended for enterprise services</li> <li>Security and bug fix release only</li> <li>Fluentd will be upgraded only in the teeny version and will stick to 1.16.x series. <ul> <li>Thus, the version number will be 5.0.x. (5.1 series will not be shipped)</li> <li>e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), v5.0.3 (Fluentd v1.16.4) ... v5.0.x (Fluentd v1.16.x)</li> </ul></li> <li>Next major version will be <code>fluent-package</code> v6.0.0, and will be shipped in 2025.</li> </ul></li> </ul> <p>As for fluent-package v5,</p> <ul> <li>fluent-package v5 LTS will be supported until March, 2025 (Even though Ruby 3.2 is still supported)</li> <li>Until 5.1.0 is released, normal release channel and LTS channel are the same package.</li> </ul> <p>Current releases and future plans:</p> <p><img src="/images/blog/20230829_fluent-package-scheduled-lifecycle.png" alt=""></p> <!-- Generated by mermaid-js: npx mmdc -t default -i fluent-package-scheduled-lifecycle.mmd -o fluent-package-scheduled-lifecycle.png gantt title Scheduled Support lifecycle for Fluent Package dateFormat YYYY-MM axisFormat %Y-%m todayMarker off section v4 %% date -d '20230508 113 days' +%Y%m%d %% => 20230829 v4.5.0 :done, v450, 2023-05-08, 113d %% date -d '20230829 124 days' +%Y%m%d %% => 20231231 v4.5.1 :active, v451, after v450, 124d section v5.x %% date -d '20230729 611 days' +%Y%m%d %% => 202350331 v5.0.0 :active, v500, 2023-07-29, 30d v5.0.1 (T.B.D.):v501, after v500, 120d v5.0.x (fluentd 1.16.x T.B.D.):v50x, after v501, 230d v5.1.x (fluentd 1.17.x T.B.D.):v51x, after v50x, 231d section v5.0 (LTS) v5.0.x (LTS) :lts_v5, 2023-07-29, 611d Fluent Pacakge v6.0.x (LTS) :milestone, lts_v6, 2025-03-31, 10d --> <table border=1 width="100%"> <tr> <th style='background-color: #eee'>Release Version</th> <th style='background-color: #eee'>Release Date</th> <th style='background-color: #eee'>End of life</th> </tr> <tr> <td>v4.5.1</td> <td>Aug, 2023</td> <td>Dec, 2023</td> </tr> <tr> <td>v5.0.1 (LTS)</td> <td>Aug 29, 2023</td> <td>March, 2025</td> </tr> <tr> <td>v6.0.0 (LTS)</td> <td>March, 2025? (T.B.D.)</td> <td>T.B.D.</td> </tr> </table> <p>Happy logging!</p> 2023-08-29 <p>Hi users!</p> <p>We have released <code>fluent-package</code> <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.0">v5.0.0</a> (Release Candidate) and the following <a href="https://github.com/fluent/fluent-package-builder/releases/tag/v5.0.1">v5.0.1</a> (General Availability). fluent-package is formerly known as <code>td-agent</code> and a stable distribution package of Fluentd.</p> <p>In this blog article, we explains why the concept of channels is introduced.</p> <h3>Why is LTS (Long Term Support) channel introduced?</h3> <p>In the previous versions, <code>td-agent</code> was shipped as quarterly releases in most cases. It sometimes contains fluentd minor upgrade occasionally, which means that not only bug or security fixes, but also contains new features. As a result, there was a case that the stability was affected unexpectedly.</p> <p>In point view of adopting fluentd (and <code>td-agent</code>) for enterprise services, there was a requirement for the distribution which contains security or bug fixes only and scheduled lifecycle of it.</p> <p>The LTS release is the answer to support them.</p> <h3>Channels for updates</h3> <p>There are two channels of support lifecycle.</p> <ul> <li>Normal release</li> <li>LTS (Long Term Support)</li> </ul> <p>Here is the difference of these channels.</p> <ul> <li>Normal release (5.x series) <ul> <li>Recommended for early adopters (eager to try new features of Fluentd)</li> <li>Rapid release development (planned semi-annually releases, but it may vary.)</li> <li>Fluentd will be upgraded occasionally <ul> <li>Minor upgrade will be applied. e.g. 1.16 =&gt; 1.17 and so on.</li> <li>e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), ... v5.1.0 (Fluentd v1.17.0)</li> </ul></li> <li>The only latest version will be supported (bug fix will not be backported into older version)</li> </ul></li> <li>LTS (Long Term Support, 5.0.x series) <ul> <li>Recommended for enterprise services</li> <li>Security and bug fix release only</li> <li>Fluentd will be upgraded only in the teeny version and will stick to 1.16.x series. <ul> <li>Thus, the version number will be 5.0.x. (5.1 series will not be shipped)</li> <li>e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), v5.0.3 (Fluentd v1.16.4) ... v5.0.x (Fluentd v1.16.x)</li> </ul></li> <li>Next major version will be <code>fluent-package</code> v6.0.0, and will be shipped in 2025.</li> </ul></li> </ul> <p>As for fluent-package v5,</p> <ul> <li>fluent-package v5 LTS will be supported until March, 2025 (Even though Ruby 3.2 is still supported)</li> <li>Until 5.1.0 is released, normal release channel and LTS channel are the same package.</li> </ul> <p>Current releases and future plans:</p> <p><img src="/images/blog/20230829_fluent-package-scheduled-lifecycle.png" alt=""></p> <!-- Generated by mermaid-js: npx mmdc -t default -i fluent-package-scheduled-lifecycle.mmd -o fluent-package-scheduled-lifecycle.png gantt title Scheduled Support lifecycle for Fluent Package dateFormat YYYY-MM axisFormat %Y-%m todayMarker off section v4 %% date -d '20230508 113 days' +%Y%m%d %% => 20230829 v4.5.0 :done, v450, 2023-05-08, 113d %% date -d '20230829 124 days' +%Y%m%d %% => 20231231 v4.5.1 :active, v451, after v450, 124d section v5.x %% date -d '20230729 611 days' +%Y%m%d %% => 202350331 v5.0.0 :active, v500, 2023-07-29, 30d v5.0.1 (T.B.D.):v501, after v500, 120d v5.0.x (fluentd 1.16.x T.B.D.):v50x, after v501, 230d v5.1.x (fluentd 1.17.x T.B.D.):v51x, after v50x, 231d section v5.0 (LTS) v5.0.x (LTS) :lts_v5, 2023-07-29, 611d Fluent Pacakge v6.0.x (LTS) :milestone, lts_v6, 2025-03-31, 10d --> <table border=1 width="100%"> <tr> <th style='background-color: #eee'>Release Version</th> <th style='background-color: #eee'>Release Date</th> <th style='background-color: #eee'>End of life</th> </tr> <tr> <td>v4.5.1</td> <td>Aug, 2023</td> <td>Dec, 2023</td> </tr> <tr> <td>v5.0.1 (LTS)</td> <td>Aug 29, 2023</td> <td>March, 2025</td> </tr> <tr> <td>v6.0.0 (LTS)</td> <td>March, 2025? (T.B.D.)</td> <td>T.B.D.</td> </tr> </table> <p>Happy logging!</p>