Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v1.9.2 has been released https://www.fluentd.org/blog/fluentd-v1.9.2-has-been-released <p>Hi users!</p> <p>We have released v1.9.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements.</p> <h3>in_tail: Add <code>pos_file_compaction_interval</code> parameter for auto compaction</h3> <p>With <code>pos_file_compaction_interval 10m</code>, in_tail removes unwatched file from <code>pos_file</code> entries at 10m intervals. This feature is for short-live and lots of containers environment.</p> <h3>Major bug fixes</h3> <ul> <li>log: serverengine&#39;s log should be formatted with the same format of fluentd</li> <li>in_http: Fix <code>NoMethodError</code> when <code>OPTIONS</code> request doesn&#39;t have <code>Origin</code> header</li> <li>parser_syslog: Improved for parsing RFC5424 structured data in <code>parser_syslog</code></li> </ul> <p>Enjoy logging!</p> 2020-02-13 <p>Hi users!</p> <p>We have released v1.9.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements.</p> <h3>in_tail: Add <code>pos_file_compaction_interval</code> parameter for auto compaction</h3> <p>With <code>pos_file_compaction_interval 10m</code>, in_tail removes unwatched file from <code>pos_file</code> entries at 10m intervals. This feature is for short-live and lots of containers environment.</p> <h3>Major bug fixes</h3> <ul> <li>log: serverengine&#39;s log should be formatted with the same format of fluentd</li> <li>in_http: Fix <code>NoMethodError</code> when <code>OPTIONS</code> request doesn&#39;t have <code>Origin</code> header</li> <li>parser_syslog: Improved for parsing RFC5424 structured data in <code>parser_syslog</code></li> </ul> <p>Enjoy logging!</p> Fluentd v1.9.1 has been released https://www.fluentd.org/blog/fluentd-v1.9.1-has-been-released <p>Hi users!</p> <p>We have released v1.9.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements.</p> <h3>http_server helper: Support HTTPS</h3> <p><code>http_server</code> helper supports HTTPS via <a href="https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-http_server#http_server_create_https_server-title-addr-port-logger-default_app-nil-tls_opts-nil-and-block">http_server_create_https_server</a> method.</p> <p>This is mainly for 3rd party plugin migration.</p> <h3>parser_syslog: Add multiline RFC5424 support</h3> <p><code>parser_syslog</code> with <code>rfc5424</code> mode can parse multiline messages now.</p> <pre><code>&lt;16&gt;1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi,\nfrom\nFluentd! </code></pre> <p>Above log can be parsed correctly.</p> <h3>out_http: Add warning for <code>retryable_response_codes</code></h3> <p>We have a plan to remove <code>503</code> from <code>retryable_response_codes</code>&#39;s default value since fluentd v2.</p> <p>If you want to keep <code>503</code>, set it explicitly in configuration.</p> <h3>Major bug fixes</h3> <ul> <li>tls: Fix TLS version handling in secure mode</li> <li>output: Add <code>EncodingError</code> to unrecoverable errors</li> </ul> <p>Enjoy logging!</p> 2020-02-01 <p>Hi users!</p> <p>We have released v1.9.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements.</p> <h3>http_server helper: Support HTTPS</h3> <p><code>http_server</code> helper supports HTTPS via <a href="https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-http_server#http_server_create_https_server-title-addr-port-logger-default_app-nil-tls_opts-nil-and-block">http_server_create_https_server</a> method.</p> <p>This is mainly for 3rd party plugin migration.</p> <h3>parser_syslog: Add multiline RFC5424 support</h3> <p><code>parser_syslog</code> with <code>rfc5424</code> mode can parse multiline messages now.</p> <pre><code>&lt;16&gt;1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi,\nfrom\nFluentd! </code></pre> <p>Above log can be parsed correctly.</p> <h3>out_http: Add warning for <code>retryable_response_codes</code></h3> <p>We have a plan to remove <code>503</code> from <code>retryable_response_codes</code>&#39;s default value since fluentd v2.</p> <p>If you want to keep <code>503</code>, set it explicitly in configuration.</p> <h3>Major bug fixes</h3> <ul> <li>tls: Fix TLS version handling in secure mode</li> <li>output: Add <code>EncodingError</code> to unrecoverable errors</li> </ul> <p>Enjoy logging!</p> Fluentd v1.9.0 has been released https://www.fluentd.org/blog/fluentd-v1.9.0-has-been-released <p>Hi users!</p> <p>We have released v1.9.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new config reload feature and some enhancements.</p> <h3>New light-weight config reload</h3> <p>We introduce new configuration reload feature. This re-creates only internal pipeline, so this is faster than existing worker-restart model.</p> <p>You can use this config reload via <code>USR2</code> signal or <code>/api/config.gracefulReload</code> RPC.</p> <p>Any feedback would be appreciated.</p> <h3>Drop ruby 2.1/2.2/2.3 support</h3> <p>See <a href="https://www.fluentd.org/blog/drop-schedule-announcement-in-2019">Drop schedule announcement in 2019</a>.</p> <h3>Support Ruby&#39;s Time class in msgpack serde</h3> <p>Fluentd uses MessagePack format for buffering data by default. This has one limitation: Can&#39;t use msgpack ext type for non primitive class.</p> <p>Since v1.9, Fluentd supports <code>Time</code> class by <code>enable_msgpack_time_support</code> parameter.</p> <pre><code>&lt;system&gt; enable_msgpack_time_support true &lt;/system&gt; </code></pre> <p>This is useful when your log contains multiple time fields.</p> <h3>Major bug fixes</h3> <ul> <li>outut/buffer: Fix stage size computation</li> <li>buffer: Disable the optimization of Metadata instance comparison on Windows to avoid lots of buffer files</li> <li>server: Ignore Errno::EHOSTUNREACH in TLS accept to avoid fluentd restart</li> </ul> <p>Enjoy logging!</p> 2020-01-22 <p>Hi users!</p> <p>We have released v1.9.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new config reload feature and some enhancements.</p> <h3>New light-weight config reload</h3> <p>We introduce new configuration reload feature. This re-creates only internal pipeline, so this is faster than existing worker-restart model.</p> <p>You can use this config reload via <code>USR2</code> signal or <code>/api/config.gracefulReload</code> RPC.</p> <p>Any feedback would be appreciated.</p> <h3>Drop ruby 2.1/2.2/2.3 support</h3> <p>See <a href="https://www.fluentd.org/blog/drop-schedule-announcement-in-2019">Drop schedule announcement in 2019</a>.</p> <h3>Support Ruby&#39;s Time class in msgpack serde</h3> <p>Fluentd uses MessagePack format for buffering data by default. This has one limitation: Can&#39;t use msgpack ext type for non primitive class.</p> <p>Since v1.9, Fluentd supports <code>Time</code> class by <code>enable_msgpack_time_support</code> parameter.</p> <pre><code>&lt;system&gt; enable_msgpack_time_support true &lt;/system&gt; </code></pre> <p>This is useful when your log contains multiple time fields.</p> <h3>Major bug fixes</h3> <ul> <li>outut/buffer: Fix stage size computation</li> <li>buffer: Disable the optimization of Metadata instance comparison on Windows to avoid lots of buffer files</li> <li>server: Ignore Errno::EHOSTUNREACH in TLS accept to avoid fluentd restart</li> </ul> <p>Enjoy logging!</p> Fluentd v1.8.1 has been released https://www.fluentd.org/blog/fluentd-v1.8.1-has-been-released <p>Hi users!</p> <p>We have released v1.8.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>out_copy: Add <code>copy_mode</code> parameter</h3> <p><code>out_copy</code> has <code>deep_copy</code> parameter but <code>deep_copy</code> is misleading name. This uses shallow copy via ruby&#39;s <code>dup</code> method and this is not fit for nested field case.</p> <p><code>copy_mode</code> resolves this problem. <code>copy_mode</code> parameter provides <code>shallow</code> and <code>deep</code>. If you need to mutate nested field after <code>out_copy</code>, you can use <code>copy_mode deep</code>. See also <a href="https://docs.fluentd.org/output/copy#copy_mode">official out_copy article</a>.</p> <p><code>deep_copy</code> parameter is now deprecated.</p> <h3>Major bug fixes</h3> <ul> <li>parser_ltsv: Prevent garbage result by checking <code>label_delimiter</code></li> </ul> <p>Enjoy logging!</p> 2019-12-26 <p>Hi users!</p> <p>We have released v1.8.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>out_copy: Add <code>copy_mode</code> parameter</h3> <p><code>out_copy</code> has <code>deep_copy</code> parameter but <code>deep_copy</code> is misleading name. This uses shallow copy via ruby&#39;s <code>dup</code> method and this is not fit for nested field case.</p> <p><code>copy_mode</code> resolves this problem. <code>copy_mode</code> parameter provides <code>shallow</code> and <code>deep</code>. If you need to mutate nested field after <code>out_copy</code>, you can use <code>copy_mode deep</code>. See also <a href="https://docs.fluentd.org/output/copy#copy_mode">official out_copy article</a>.</p> <p><code>deep_copy</code> parameter is now deprecated.</p> <h3>Major bug fixes</h3> <ul> <li>parser_ltsv: Prevent garbage result by checking <code>label_delimiter</code></li> </ul> <p>Enjoy logging!</p> Fluentd v1.8.0 has been released https://www.fluentd.org/blog/fluentd-v1.8.0-has-been-released <p>Hi users!</p> <p>We have released v1.8.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new service discovery plugin and lots of enhancements.</p> <h3>Add service discovery plugin</h3> <p>The service discovery plugin manages the list of destination by extensible way. Here is <code>out_forward</code> example with service discovery plugin:</p> <pre><code># out_forward now uses service discovery plugin &lt;match test&gt; @type forward &lt;service_discovery&gt; @type file path /path/to/sd_forward.yaml &lt;/service_discovery&gt; # other parameters... &lt;/match&gt; # content of sd_forward.yaml - &#39;host&#39;: 127.0.0.1 &#39;port&#39;: 24224 &#39;weight&#39;: 1 &#39;name&#39;: server1 - &#39;host&#39;: 127.0.0.1 &#39;port&#39;: 24225 &#39;weight&#39;: 1 &#39;name&#39;: server2 </code></pre> <p>With this configuration, <code>out_forward</code> uses <code>file</code> service discovery plugin to read the server list from <code>sd_forward.yaml</code> file.</p> <p>In addition, <code>file</code> service discovery monitors <code>path</code> file. If <code>sd_forward.yaml</code> file is updated, the server list of <code>out_forward</code> is also updated. No need to restart fluentd for the server list of <code>out_forward</code>.</p> <p>Supported built-in plugins are now <a href="https://docs.fluentd.org/service_discovery/static">static</a> and <a href="https://docs.fluentd.org/service_discovery/file">file</a>. We have a plan to support more service discovery plugins, e.g. DNS SRV record.</p> <h3>Improve config parameter handling</h3> <p>Current configuration parser has several problems:</p> <ul> <li>Automatic conversion for non string type</li> </ul> <p>This is error prone. For example, if you set <code>int_param &quot;foo&quot;</code>, <code>int_param</code> becomes <code>0</code>. This case should raise configuration error.</p> <ul> <li>No way to use default value</li> </ul> <p>This is critical when you use environment variable. We can&#39;t use <code>nil</code> or plugin&#39;s default value with <code>param &quot;#{ENV[&#39;FOO_PARAM&#39;]}&quot;</code> case.</p> <p>For former case, we introduce <code>--strict-config-value</code> option for strict mode. With this option, fluentd raises configuration error for invalid parameter. You can also use <code>&lt;system&gt;</code> section instead of <code>--strict-config-value</code>.</p> <pre><code>&lt;system&gt; strict_config_value true &lt;/system&gt; </code></pre> <p>Fluentd v2 will use strict mode.</p> <p>For latter case, we introduce helpers for ruby embedded parameter. You can use default value like below:</p> <pre><code>param &quot;#{ENV[&#39;FOO_PARAM&#39;] || use_default}&quot; </code></pre> <p>If <code>FOO_PARAM</code> envvar doesn&#39;t exist, plugin&#39;s default will be used. This is useful on kubernetes environment.</p> <h3>formatter_csv: Support nested fields</h3> <p>You can use record accessor syntax for nested fields in <code>fields</code> parameter.</p> <pre><code>&lt;format&gt; @type csv fields code,message,$.nest.key &lt;/format&gt; </code></pre> <h3>Deprecate top-level match for capturing fluentd logs</h3> <p>Fluentd now provides 2 approaches to capture fluentd logs.</p> <ul> <li><code>&lt;label @FLUENT_LOG&gt;</code> with <code>&lt;filter&gt;</code> and <code>&lt;match&gt;</code></li> <li>top level <code>&lt;match fluent.**&gt;</code></li> </ul> <p>Former is v1 style. Latter is for v0.12 compatibility. Fluentd now shows deprecation message for latter case.</p> <p>Fluentd v2 will remove top level <code>&lt;match fluent.**&gt;</code> feature for fluentd logs.</p> <h3>Major bug fixes</h3> <ul> <li>Fix error handling during dry-run mode. Show better logs</li> <li>server helper: Fix IPv6 dual stack mode issue for tcp socket</li> <li>server helper: Ignore <code>ECONNREFUSED</code> in TLS accept to avoid fluentd restart</li> </ul> <p>Enjoy logging!</p> 2019-12-12 <p>Hi users!</p> <p>We have released v1.8.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new service discovery plugin and lots of enhancements.</p> <h3>Add service discovery plugin</h3> <p>The service discovery plugin manages the list of destination by extensible way. Here is <code>out_forward</code> example with service discovery plugin:</p> <pre><code># out_forward now uses service discovery plugin &lt;match test&gt; @type forward &lt;service_discovery&gt; @type file path /path/to/sd_forward.yaml &lt;/service_discovery&gt; # other parameters... &lt;/match&gt; # content of sd_forward.yaml - &#39;host&#39;: 127.0.0.1 &#39;port&#39;: 24224 &#39;weight&#39;: 1 &#39;name&#39;: server1 - &#39;host&#39;: 127.0.0.1 &#39;port&#39;: 24225 &#39;weight&#39;: 1 &#39;name&#39;: server2 </code></pre> <p>With this configuration, <code>out_forward</code> uses <code>file</code> service discovery plugin to read the server list from <code>sd_forward.yaml</code> file.</p> <p>In addition, <code>file</code> service discovery monitors <code>path</code> file. If <code>sd_forward.yaml</code> file is updated, the server list of <code>out_forward</code> is also updated. No need to restart fluentd for the server list of <code>out_forward</code>.</p> <p>Supported built-in plugins are now <a href="https://docs.fluentd.org/service_discovery/static">static</a> and <a href="https://docs.fluentd.org/service_discovery/file">file</a>. We have a plan to support more service discovery plugins, e.g. DNS SRV record.</p> <h3>Improve config parameter handling</h3> <p>Current configuration parser has several problems:</p> <ul> <li>Automatic conversion for non string type</li> </ul> <p>This is error prone. For example, if you set <code>int_param &quot;foo&quot;</code>, <code>int_param</code> becomes <code>0</code>. This case should raise configuration error.</p> <ul> <li>No way to use default value</li> </ul> <p>This is critical when you use environment variable. We can&#39;t use <code>nil</code> or plugin&#39;s default value with <code>param &quot;#{ENV[&#39;FOO_PARAM&#39;]}&quot;</code> case.</p> <p>For former case, we introduce <code>--strict-config-value</code> option for strict mode. With this option, fluentd raises configuration error for invalid parameter. You can also use <code>&lt;system&gt;</code> section instead of <code>--strict-config-value</code>.</p> <pre><code>&lt;system&gt; strict_config_value true &lt;/system&gt; </code></pre> <p>Fluentd v2 will use strict mode.</p> <p>For latter case, we introduce helpers for ruby embedded parameter. You can use default value like below:</p> <pre><code>param &quot;#{ENV[&#39;FOO_PARAM&#39;] || use_default}&quot; </code></pre> <p>If <code>FOO_PARAM</code> envvar doesn&#39;t exist, plugin&#39;s default will be used. This is useful on kubernetes environment.</p> <h3>formatter_csv: Support nested fields</h3> <p>You can use record accessor syntax for nested fields in <code>fields</code> parameter.</p> <pre><code>&lt;format&gt; @type csv fields code,message,$.nest.key &lt;/format&gt; </code></pre> <h3>Deprecate top-level match for capturing fluentd logs</h3> <p>Fluentd now provides 2 approaches to capture fluentd logs.</p> <ul> <li><code>&lt;label @FLUENT_LOG&gt;</code> with <code>&lt;filter&gt;</code> and <code>&lt;match&gt;</code></li> <li>top level <code>&lt;match fluent.**&gt;</code></li> </ul> <p>Former is v1 style. Latter is for v0.12 compatibility. Fluentd now shows deprecation message for latter case.</p> <p>Fluentd v2 will remove top level <code>&lt;match fluent.**&gt;</code> feature for fluentd logs.</p> <h3>Major bug fixes</h3> <ul> <li>Fix error handling during dry-run mode. Show better logs</li> <li>server helper: Fix IPv6 dual stack mode issue for tcp socket</li> <li>server helper: Ignore <code>ECONNREFUSED</code> in TLS accept to avoid fluentd restart</li> </ul> <p>Enjoy logging!</p>