Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v0.14.19 has been released https://www.fluentd.org/blog/fluentd-v0.14.19-has-been-released <p>Hi users!</p> <p>We have released v0.14.19. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>): This release is mainly for bug fix and several plugin improvement.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>filter_grep: New <code>&lt;regexp&gt;</code>/<code>&lt;exclude&gt;</code> section</h3> <p>We introduce <code>&lt;regexp&gt;</code>/<code>&lt;exclude&gt;</code> section to <code>filter_grep</code>. Here is an example:</p> <pre><code>&lt;filter pattern&gt; @type grep &lt;regexp&gt; key message pattern cool &lt;/regexp&gt; &lt;regexp&gt; key hostname pattern ^web\d+\.example\.com$ &lt;/regexp&gt; &lt;/filter&gt; </code></pre> <p>This is same as:</p> <pre><code>&lt;filter pattern&gt; @type grep regexp1 message cool regexp2 hostname ^web\d+\.example\.com$ &lt;/filter&gt; </code></pre> <p>New section approach is easy to parse by fluentd core and we have a plan to add nested record access support via new section parameters.</p> <h3>in_syslog improvement</h3> <ul> <li>Allow more characters in identifier part</li> </ul> <p>Old implementation allows only <code>[a-zA-Z0-9_\/\.\-]</code> characters in identifier part. This follows RFC3614 but real world systems use more characters. For the real world, <code>in_syslog</code> now accept more characters except <code></code>, <code>:</code> and <code>[</code>.</p> <ul> <li>Add <code>resolve_hostname</code> parameter</li> </ul> <p>This parameter is useful for reducing resolve cost when you don&#39;t use <code>source_hostname_key</code>.</p> <h3>Major bug fixes</h3> <ul> <li>output: Fix race condition during retry</li> </ul> <p>Users sometimes hit <code>&quot;undefined method `next_time&#39; for nil:NilClass&quot;</code> error inside flush thread of output plugins. v0.14.19 fixed it.</p> <p>We also released v0.12.38. This is for adding v0.14.19 features to <code>in_syslog</code> and <code>filter_grep</code> to reduce the differences between v0.12 and v0.14.</p> <p>Enjoy logging!</p> 2017-07-13 <p>Hi users!</p> <p>We have released v0.14.19. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>): This release is mainly for bug fix and several plugin improvement.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>filter_grep: New <code>&lt;regexp&gt;</code>/<code>&lt;exclude&gt;</code> section</h3> <p>We introduce <code>&lt;regexp&gt;</code>/<code>&lt;exclude&gt;</code> section to <code>filter_grep</code>. Here is an example:</p> <pre><code>&lt;filter pattern&gt; @type grep &lt;regexp&gt; key message pattern cool &lt;/regexp&gt; &lt;regexp&gt; key hostname pattern ^web\d+\.example\.com$ &lt;/regexp&gt; &lt;/filter&gt; </code></pre> <p>This is same as:</p> <pre><code>&lt;filter pattern&gt; @type grep regexp1 message cool regexp2 hostname ^web\d+\.example\.com$ &lt;/filter&gt; </code></pre> <p>New section approach is easy to parse by fluentd core and we have a plan to add nested record access support via new section parameters.</p> <h3>in_syslog improvement</h3> <ul> <li>Allow more characters in identifier part</li> </ul> <p>Old implementation allows only <code>[a-zA-Z0-9_\/\.\-]</code> characters in identifier part. This follows RFC3614 but real world systems use more characters. For the real world, <code>in_syslog</code> now accept more characters except <code></code>, <code>:</code> and <code>[</code>.</p> <ul> <li>Add <code>resolve_hostname</code> parameter</li> </ul> <p>This parameter is useful for reducing resolve cost when you don&#39;t use <code>source_hostname_key</code>.</p> <h3>Major bug fixes</h3> <ul> <li>output: Fix race condition during retry</li> </ul> <p>Users sometimes hit <code>&quot;undefined method `next_time&#39; for nil:NilClass&quot;</code> error inside flush thread of output plugins. v0.14.19 fixed it.</p> <p>We also released v0.12.38. This is for adding v0.14.19 features to <code>in_syslog</code> and <code>filter_grep</code> to reduce the differences between v0.12 and v0.14.</p> <p>Enjoy logging!</p> Fluentd v0.14.17 has been released https://www.fluentd.org/blog/fluentd-v0.14.17-has-been-released <p>Hi users!</p> <p>We have released v0.14.17. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>): This release is mainly for bug fixes.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>server: Accept private key for TLS server without passphrase</h3> <p>The <code>private_key_passphrase</code> and <code>ca_private_key_passphrase</code> for TLS setting are now optional. It means you can use private key without passphrase in server plugin helper based plugins, e.g. <code>in_forward</code>/<code>out_forward</code>. </p> <h3>Major bug fixes</h3> <ul> <li>out_forward: Fix ack handling</li> <li>config: Mask all secret parameters in worker section</li> </ul> <p>This version fixed <code>out_forward</code>&#39;s <code>require_ack_response</code> regressions. If you use <code>require_ack_response</code> with v0.14, we recommend to update fluentd v0.14 to v0.14.17.</p> <p>Enjoy logging!</p> 2017-05-30 <p>Hi users!</p> <p>We have released v0.14.17. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>): This release is mainly for bug fixes.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>server: Accept private key for TLS server without passphrase</h3> <p>The <code>private_key_passphrase</code> and <code>ca_private_key_passphrase</code> for TLS setting are now optional. It means you can use private key without passphrase in server plugin helper based plugins, e.g. <code>in_forward</code>/<code>out_forward</code>. </p> <h3>Major bug fixes</h3> <ul> <li>out_forward: Fix ack handling</li> <li>config: Mask all secret parameters in worker section</li> </ul> <p>This version fixed <code>out_forward</code>&#39;s <code>require_ack_response</code> regressions. If you use <code>require_ack_response</code> with v0.14, we recommend to update fluentd v0.14 to v0.14.17.</p> <p>Enjoy logging!</p> Fluentd v0.14.16 has been released https://www.fluentd.org/blog/fluentd-v0.14.16-has-been-released <p>Hi users!</p> <p>We have released v0.14.16. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>): This release is mainly for fixing regressions.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>Allow null character in double-quoted string</h3> <p>You can use null character <code>\0</code> with double-quoted string. This is useful for delimiter, e.g. tcp input, tsv parser, etc. Here is an example:</p> <pre><code>&lt;source&gt; @type tail path /path/to/app.log tag tail.test format tsv keys a,b,c delimiter &quot;\0&quot; # \0 or &#39;\0&#39; doesn&#39;t work &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>out_forward: Call proper method for each connection type</li> <li>in_monitor_agent: check variable buffer is a Buffer instance</li> </ul> <p>Enjoy logging!</p> 2017-05-14 <p>Hi users!</p> <p>We have released v0.14.16. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>): This release is mainly for fixing regressions.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>Allow null character in double-quoted string</h3> <p>You can use null character <code>\0</code> with double-quoted string. This is useful for delimiter, e.g. tcp input, tsv parser, etc. Here is an example:</p> <pre><code>&lt;source&gt; @type tail path /path/to/app.log tag tail.test format tsv keys a,b,c delimiter &quot;\0&quot; # \0 or &#39;\0&#39; doesn&#39;t work &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>out_forward: Call proper method for each connection type</li> <li>in_monitor_agent: check variable buffer is a Buffer instance</li> </ul> <p>Enjoy logging!</p> Fluentd v0.14.15 has been released https://www.fluentd.org/blog/fluentd-v0.14.15-has-been-released <p>Hi users!</p> <p>We have released v0.14.15. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3><code>&lt;worker N&gt;</code> directive</h3> <p>We introduced built-in multi-process feature since v0.14.12(See <a href="http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released">release announcement</a>). New <code>&lt;worker N&gt;</code> directive enhances multi-process feature.</p> <p>The limitation of multi-process was non-multi-process ready plugin can&#39;t be used together. For example, we can&#39;t use in_tail plugin with <code>workers</code> parameter. Following configuration raises an error at startup.</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type forward &lt;/source&gt; &lt;source&gt; @type tail &lt;/source&gt; </code></pre> <p><code>&lt;worker N&gt;</code> directive resolves this problem. You can invoke any plugins in specific worker(<code>N</code> is worker index) and plugins under <code>&lt;worker N&gt;</code> acts as if they work on single process environment. We can mix multi-process ready and non-multi-process ready plugins in one instance. Working example with above configuration is like below:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; # work on multi process environment. worker0 - worker3 invoke in_forward plugin &lt;source&gt; @type forward &lt;/source&gt; # work on only worker 0. worker1 - worker3 don&#39;t invoke in_tail plugin &lt;worker 0&gt; &lt;source&gt; @type tail &lt;/source&gt; &lt;/worker&gt; # &lt;worker 1&gt;, &lt;worker 2&gt; or &lt;worker 3&gt; is also ok </code></pre> <h3>Plugin: Add <code>formatted_to_msgpack_binary?</code> method to Output plugin API</h3> <p>We use <code>formatted_to_msgpack_binary</code> method to tell &quot;This plugin uses msgpack for event serialization&quot; to output core when you customize <code>format</code> method. This name is not rubyish, so we add <code>formatted_to_msgpack_binary?</code> method for better name. We recommend to use <code>formatted_to_msgpack_binary?</code> for new plugins.</p> <p>Of course, <code>formatted_to_msgpack_binary</code> method is available for existing plugins.</p> <h3>Windows: Add several options to configure Fluentd windows service information</h3> <p>By default, Fluentd uses following settings for Windows service registration:</p> <ul> <li>Service name: <code>fluentdwinsvc</code></li> <li>Display name: <code>Fluentd Windows Service</code></li> <li>Service description: <code>Fluentd is an event collector system.</code></li> </ul> <p>Since v0.14.15, you can configure these settings via command line options.</p> <ul> <li>Service name: <code>--winsvc-name NAME</code></li> <li>Display name: <code>--winsvc-display-name DISPLAY_NAME</code></li> <li>Service description: <code>--winsvc-desc DESC</code></li> </ul> <h3>Major bug fixes</h3> <ul> <li>in_http: Fix X-Forwarded-For header handling. Accpet multiple headers</li> <li>windows: Quote the file path to the Ruby bin directory when starting fluentd as a windows service</li> </ul> <p>Enjoy logging!</p> 2017-04-24 <p>Hi users!</p> <p>We have released v0.14.15. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3><code>&lt;worker N&gt;</code> directive</h3> <p>We introduced built-in multi-process feature since v0.14.12(See <a href="http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released">release announcement</a>). New <code>&lt;worker N&gt;</code> directive enhances multi-process feature.</p> <p>The limitation of multi-process was non-multi-process ready plugin can&#39;t be used together. For example, we can&#39;t use in_tail plugin with <code>workers</code> parameter. Following configuration raises an error at startup.</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type forward &lt;/source&gt; &lt;source&gt; @type tail &lt;/source&gt; </code></pre> <p><code>&lt;worker N&gt;</code> directive resolves this problem. You can invoke any plugins in specific worker(<code>N</code> is worker index) and plugins under <code>&lt;worker N&gt;</code> acts as if they work on single process environment. We can mix multi-process ready and non-multi-process ready plugins in one instance. Working example with above configuration is like below:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; # work on multi process environment. worker0 - worker3 invoke in_forward plugin &lt;source&gt; @type forward &lt;/source&gt; # work on only worker 0. worker1 - worker3 don&#39;t invoke in_tail plugin &lt;worker 0&gt; &lt;source&gt; @type tail &lt;/source&gt; &lt;/worker&gt; # &lt;worker 1&gt;, &lt;worker 2&gt; or &lt;worker 3&gt; is also ok </code></pre> <h3>Plugin: Add <code>formatted_to_msgpack_binary?</code> method to Output plugin API</h3> <p>We use <code>formatted_to_msgpack_binary</code> method to tell &quot;This plugin uses msgpack for event serialization&quot; to output core when you customize <code>format</code> method. This name is not rubyish, so we add <code>formatted_to_msgpack_binary?</code> method for better name. We recommend to use <code>formatted_to_msgpack_binary?</code> for new plugins.</p> <p>Of course, <code>formatted_to_msgpack_binary</code> method is available for existing plugins.</p> <h3>Windows: Add several options to configure Fluentd windows service information</h3> <p>By default, Fluentd uses following settings for Windows service registration:</p> <ul> <li>Service name: <code>fluentdwinsvc</code></li> <li>Display name: <code>Fluentd Windows Service</code></li> <li>Service description: <code>Fluentd is an event collector system.</code></li> </ul> <p>Since v0.14.15, you can configure these settings via command line options.</p> <ul> <li>Service name: <code>--winsvc-name NAME</code></li> <li>Display name: <code>--winsvc-display-name DISPLAY_NAME</code></li> <li>Service description: <code>--winsvc-desc DESC</code></li> </ul> <h3>Major bug fixes</h3> <ul> <li>in_http: Fix X-Forwarded-For header handling. Accpet multiple headers</li> <li>windows: Quote the file path to the Ruby bin directory when starting fluentd as a windows service</li> </ul> <p>Enjoy logging!</p> Fluentd v0.14.14 has been released https://www.fluentd.org/blog/fluentd-v0.14.14-has-been-released <p>Hi users!</p> <p>We have released v0.14.14 with several bug fixes and plugin improvement.</p> <p>Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>in_monitor_agent: Start one HTTP server per worker on sequential port numbers</h3> <p><code>in_monitor_agent</code> now launches HTTP server per worker on multiprocess environment. The port number is based on <code>port</code> parameter and worker id. For example, if you have following configuration:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type monitor_agent port 25000 &lt;/source&gt; </code></pre> <p><code>worker 1</code>&#39;s port is 25000, <code>worker 2</code>&#39;s port is 25001 and <code>worker 4</code>&#39;s port is 25003.</p> <h3>in_http: Support <code>application/msgpack</code> header</h3> <p><a href="http://fluentbit.io/">fluent-bit</a> has <code>out_http</code> and it uses <code>application/json</code> or <code>application/msgpack</code> header in the request. fluentd&#39;s <code>in_http</code> now accepts <code>application/msgpack</code> request from fluent-bit :)</p> <h3>filter_parser: Add <code>emit_invalid_record_to_error</code> parameter</h3> <p>By default, <code>filter_parser</code> emits invalid record to <code>@ERROR</code> label when parsing failed. This is useful for rescuing invalid records in your data pipeline. But users sometimes want to ignore such records for mixed logs. You can now ignore unparseable records by setting <code>emit_invalid_record_to_error false</code> in the configuration.</p> <pre><code>&lt;filter pattern&gt; @type parser # ignore non-json logs emit_invalid_record_to_error false &lt;parse&gt; @format json &lt;/parse&gt; &lt;/filter&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>config: Set encoding forcefully to avoid UndefinedConversionError</li> <li>config: Fix <code>log_level</code> handling in <system></li> <li>Fix Input and Output deadlock when buffer is full during startup</li> </ul> <p>Enjoy logging!</p> 2017-03-24 <p>Hi users!</p> <p>We have released v0.14.14 with several bug fixes and plugin improvement.</p> <p>Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>in_monitor_agent: Start one HTTP server per worker on sequential port numbers</h3> <p><code>in_monitor_agent</code> now launches HTTP server per worker on multiprocess environment. The port number is based on <code>port</code> parameter and worker id. For example, if you have following configuration:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type monitor_agent port 25000 &lt;/source&gt; </code></pre> <p><code>worker 1</code>&#39;s port is 25000, <code>worker 2</code>&#39;s port is 25001 and <code>worker 4</code>&#39;s port is 25003.</p> <h3>in_http: Support <code>application/msgpack</code> header</h3> <p><a href="http://fluentbit.io/">fluent-bit</a> has <code>out_http</code> and it uses <code>application/json</code> or <code>application/msgpack</code> header in the request. fluentd&#39;s <code>in_http</code> now accepts <code>application/msgpack</code> request from fluent-bit :)</p> <h3>filter_parser: Add <code>emit_invalid_record_to_error</code> parameter</h3> <p>By default, <code>filter_parser</code> emits invalid record to <code>@ERROR</code> label when parsing failed. This is useful for rescuing invalid records in your data pipeline. But users sometimes want to ignore such records for mixed logs. You can now ignore unparseable records by setting <code>emit_invalid_record_to_error false</code> in the configuration.</p> <pre><code>&lt;filter pattern&gt; @type parser # ignore non-json logs emit_invalid_record_to_error false &lt;parse&gt; @format json &lt;/parse&gt; &lt;/filter&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>config: Set encoding forcefully to avoid UndefinedConversionError</li> <li>config: Fix <code>log_level</code> handling in <system></li> <li>Fix Input and Output deadlock when buffer is full during startup</li> </ul> <p>Enjoy logging!</p>