Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v0.14.20 has been released https://www.fluentd.org/blog/fluentd-v0.14.20-has-been-released <p>Hi users!</p> <p>We have released v0.14.20. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>): This release is mainly for bug fixes and new plugin helper.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>record_accessor helper: New plugin helper for nested record access</h3> <p>We introduce new <code>record_accessor</code> helper. This provides unified nested record access mechanism for plugins. Some plugins don&#39;t support nested record access and it makes the configuration complicated. <code>record_accessor</code> helper mitigates this problem.</p> <p>Here is an example of grep filter plugin with <code>record_accessor</code>:</p> <pre><code>&lt;filter pattern&gt; @type grep &lt;regexp&gt; key $.event.level # Access record[&quot;event&quot;][&quot;level&quot;] field, not record[&quot;$.event.level&quot;] pattern WARN &lt;/regexp&gt; &lt;/filter&gt; </code></pre> <p><code>record_accessor</code> supports two jsonpath like syntax:</p> <ul> <li>dot notation: <code>$.</code> started parameter. Chain fields by <code>.</code></li> </ul> <p>Simple syntax: <code>$.event.level</code> for <code>record[&quot;event&quot;][&quot;level&quot;]</code>, <code>$.key1[0].key2</code> for <code>record[&quot;key1&quot;][0][&quot;key2&quot;]</code></p> <ul> <li>bracket notation: <code>$[</code> started parameter. Chain fields by <code>[]</code></li> </ul> <p>Useful for special characters, <code>.</code>, <code></code> and etc: <code>$[&quot;dot.key&quot;][0][&quot;space key&quot;]</code> for <code>record[&quot;dot.key&quot;][0][&quot;space key&quot;]</code></p> <p>If you set non <code>$.</code> or <code>$[</code> started value, e.g. <code>key log</code>, it is same as <code>record[&quot;log&quot;]</code>. So using <code>record_accessor</code> doesn&#39;t break existing plugin behaviour.</p> <p>To plugin developers, this is code example of plugin:</p> <pre><code class="rb">require &#39;fluent/plugin/filter&#39; module Fluent::Plugin class FooFilter &lt; Filter Fluent::Plugin.register_filter(&#39;foo&#39;, self) # 1. load record_accessor helper helpers :record_accessor def configure(conf) # 2. Call `record_accessor_create` to create accessor object. # Actually, &quot;$.key1.key2&quot; comes from the configuration @accessor = record_accessor_create(&quot;$.key1.key2&quot;) end def filter(tag, time, record) # 3. Call `call` method value = @accessor.call(record) # Get record[&quot;key1&quot;][&quot;key2&quot;] field value # ... end end end </code></pre> <p>Currently, <code>filter_grep</code> supports <code>record_accessor</code>. We will add <code>record_accessor</code> support to other plugins.</p> <h3>Fluentd log configuration: Add <log> in <system> directive</h3> <p>We sometimes got the request &quot;We want fluentd&#39;s log as json format like Docker. json is easy to parse.&quot; This is good idea, so we add <code>&lt;log&gt;</code> directive to under <code>&lt;system&gt;</code> directive.</p> <pre><code>&lt;system&gt; &lt;log&gt; # text or json. default is text format json # Change format of log time. This affects both text and json. time_format FT%T%:z &lt;/log&gt; &lt;/system&gt; </code></pre> <p>Here are log example:</p> <ul> <li>text</li> </ul> <pre><code>2017-07-27 07:37:02 +0900 [info]: #0 fluentd worker is now running worker=0 </code></pre> <ul> <li>json</li> </ul> <pre><code>{&quot;time&quot;:&quot;2017-07-27 06:44:54 +0900&quot;,&quot;level&quot;:&quot;info&quot;,&quot;message&quot;:&quot;fluentd worker is now running worker=0&quot;,&quot;worker_id&quot;:0} </code></pre> <h3>Major bug fixes</h3> <ul> <li>buf_file: Improve file handling to mitigate broken meta file</li> </ul> <p>We got a bug report about broken meta file with <code>buf_file</code>. v0.14.20 includes the patch for this problem. We need the feedback the problem still happens or not.</p> <ul> <li>process: Send signal to all workers</li> </ul> <p>Before, Supervisor sends a signal to latest one on multiple process environment. Supervisor now sends a signal to all workers. </p> <p>Enjoy logging!</p> 2017-08-01 <p>Hi users!</p> <p>We have released v0.14.20. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>): This release is mainly for bug fixes and new plugin helper.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>record_accessor helper: New plugin helper for nested record access</h3> <p>We introduce new <code>record_accessor</code> helper. This provides unified nested record access mechanism for plugins. Some plugins don&#39;t support nested record access and it makes the configuration complicated. <code>record_accessor</code> helper mitigates this problem.</p> <p>Here is an example of grep filter plugin with <code>record_accessor</code>:</p> <pre><code>&lt;filter pattern&gt; @type grep &lt;regexp&gt; key $.event.level # Access record[&quot;event&quot;][&quot;level&quot;] field, not record[&quot;$.event.level&quot;] pattern WARN &lt;/regexp&gt; &lt;/filter&gt; </code></pre> <p><code>record_accessor</code> supports two jsonpath like syntax:</p> <ul> <li>dot notation: <code>$.</code> started parameter. Chain fields by <code>.</code></li> </ul> <p>Simple syntax: <code>$.event.level</code> for <code>record[&quot;event&quot;][&quot;level&quot;]</code>, <code>$.key1[0].key2</code> for <code>record[&quot;key1&quot;][0][&quot;key2&quot;]</code></p> <ul> <li>bracket notation: <code>$[</code> started parameter. Chain fields by <code>[]</code></li> </ul> <p>Useful for special characters, <code>.</code>, <code></code> and etc: <code>$[&quot;dot.key&quot;][0][&quot;space key&quot;]</code> for <code>record[&quot;dot.key&quot;][0][&quot;space key&quot;]</code></p> <p>If you set non <code>$.</code> or <code>$[</code> started value, e.g. <code>key log</code>, it is same as <code>record[&quot;log&quot;]</code>. So using <code>record_accessor</code> doesn&#39;t break existing plugin behaviour.</p> <p>To plugin developers, this is code example of plugin:</p> <pre><code class="rb">require &#39;fluent/plugin/filter&#39; module Fluent::Plugin class FooFilter &lt; Filter Fluent::Plugin.register_filter(&#39;foo&#39;, self) # 1. load record_accessor helper helpers :record_accessor def configure(conf) # 2. Call `record_accessor_create` to create accessor object. # Actually, &quot;$.key1.key2&quot; comes from the configuration @accessor = record_accessor_create(&quot;$.key1.key2&quot;) end def filter(tag, time, record) # 3. Call `call` method value = @accessor.call(record) # Get record[&quot;key1&quot;][&quot;key2&quot;] field value # ... end end end </code></pre> <p>Currently, <code>filter_grep</code> supports <code>record_accessor</code>. We will add <code>record_accessor</code> support to other plugins.</p> <h3>Fluentd log configuration: Add <log> in <system> directive</h3> <p>We sometimes got the request &quot;We want fluentd&#39;s log as json format like Docker. json is easy to parse.&quot; This is good idea, so we add <code>&lt;log&gt;</code> directive to under <code>&lt;system&gt;</code> directive.</p> <pre><code>&lt;system&gt; &lt;log&gt; # text or json. default is text format json # Change format of log time. This affects both text and json. time_format FT%T%:z &lt;/log&gt; &lt;/system&gt; </code></pre> <p>Here are log example:</p> <ul> <li>text</li> </ul> <pre><code>2017-07-27 07:37:02 +0900 [info]: #0 fluentd worker is now running worker=0 </code></pre> <ul> <li>json</li> </ul> <pre><code>{&quot;time&quot;:&quot;2017-07-27 06:44:54 +0900&quot;,&quot;level&quot;:&quot;info&quot;,&quot;message&quot;:&quot;fluentd worker is now running worker=0&quot;,&quot;worker_id&quot;:0} </code></pre> <h3>Major bug fixes</h3> <ul> <li>buf_file: Improve file handling to mitigate broken meta file</li> </ul> <p>We got a bug report about broken meta file with <code>buf_file</code>. v0.14.20 includes the patch for this problem. We need the feedback the problem still happens or not.</p> <ul> <li>process: Send signal to all workers</li> </ul> <p>Before, Supervisor sends a signal to latest one on multiple process environment. Supervisor now sends a signal to all workers. </p> <p>Enjoy logging!</p> Fluentd v0.14.19 has been released https://www.fluentd.org/blog/fluentd-v0.14.19-has-been-released <p>Hi users!</p> <p>We have released v0.14.19. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>): This release is mainly for bug fix and several plugin improvement.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>filter_grep: New <code>&lt;regexp&gt;</code>/<code>&lt;exclude&gt;</code> section</h3> <p>We introduce <code>&lt;regexp&gt;</code>/<code>&lt;exclude&gt;</code> section to <code>filter_grep</code>. Here is an example:</p> <pre><code>&lt;filter pattern&gt; @type grep &lt;regexp&gt; key message pattern cool &lt;/regexp&gt; &lt;regexp&gt; key hostname pattern ^web\d+\.example\.com$ &lt;/regexp&gt; &lt;/filter&gt; </code></pre> <p>This is same as:</p> <pre><code>&lt;filter pattern&gt; @type grep regexp1 message cool regexp2 hostname ^web\d+\.example\.com$ &lt;/filter&gt; </code></pre> <p>New section approach is easy to parse by fluentd core and we have a plan to add nested record access support via new section parameters.</p> <h3>in_syslog improvement</h3> <ul> <li>Allow more characters in identifier part</li> </ul> <p>Old implementation allows only <code>[a-zA-Z0-9_\/\.\-]</code> characters in identifier part. This follows RFC3614 but real world systems use more characters. For the real world, <code>in_syslog</code> now accept more characters except <code></code>, <code>:</code> and <code>[</code>.</p> <ul> <li>Add <code>resolve_hostname</code> parameter</li> </ul> <p>This parameter is useful for reducing resolve cost when you don&#39;t use <code>source_hostname_key</code>.</p> <h3>Major bug fixes</h3> <ul> <li>output: Fix race condition during retry</li> </ul> <p>Users sometimes hit <code>&quot;undefined method `next_time&#39; for nil:NilClass&quot;</code> error inside flush thread of output plugins. v0.14.19 fixed it.</p> <p>We also released v0.12.38. This is for adding v0.14.19 features to <code>in_syslog</code> and <code>filter_grep</code> to reduce the differences between v0.12 and v0.14.</p> <p>Enjoy logging!</p> 2017-07-13 <p>Hi users!</p> <p>We have released v0.14.19. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>): This release is mainly for bug fix and several plugin improvement.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>filter_grep: New <code>&lt;regexp&gt;</code>/<code>&lt;exclude&gt;</code> section</h3> <p>We introduce <code>&lt;regexp&gt;</code>/<code>&lt;exclude&gt;</code> section to <code>filter_grep</code>. Here is an example:</p> <pre><code>&lt;filter pattern&gt; @type grep &lt;regexp&gt; key message pattern cool &lt;/regexp&gt; &lt;regexp&gt; key hostname pattern ^web\d+\.example\.com$ &lt;/regexp&gt; &lt;/filter&gt; </code></pre> <p>This is same as:</p> <pre><code>&lt;filter pattern&gt; @type grep regexp1 message cool regexp2 hostname ^web\d+\.example\.com$ &lt;/filter&gt; </code></pre> <p>New section approach is easy to parse by fluentd core and we have a plan to add nested record access support via new section parameters.</p> <h3>in_syslog improvement</h3> <ul> <li>Allow more characters in identifier part</li> </ul> <p>Old implementation allows only <code>[a-zA-Z0-9_\/\.\-]</code> characters in identifier part. This follows RFC3614 but real world systems use more characters. For the real world, <code>in_syslog</code> now accept more characters except <code></code>, <code>:</code> and <code>[</code>.</p> <ul> <li>Add <code>resolve_hostname</code> parameter</li> </ul> <p>This parameter is useful for reducing resolve cost when you don&#39;t use <code>source_hostname_key</code>.</p> <h3>Major bug fixes</h3> <ul> <li>output: Fix race condition during retry</li> </ul> <p>Users sometimes hit <code>&quot;undefined method `next_time&#39; for nil:NilClass&quot;</code> error inside flush thread of output plugins. v0.14.19 fixed it.</p> <p>We also released v0.12.38. This is for adding v0.14.19 features to <code>in_syslog</code> and <code>filter_grep</code> to reduce the differences between v0.12 and v0.14.</p> <p>Enjoy logging!</p> Fluentd v0.14.17 has been released https://www.fluentd.org/blog/fluentd-v0.14.17-has-been-released <p>Hi users!</p> <p>We have released v0.14.17. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>): This release is mainly for bug fixes.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>server: Accept private key for TLS server without passphrase</h3> <p>The <code>private_key_passphrase</code> and <code>ca_private_key_passphrase</code> for TLS setting are now optional. It means you can use private key without passphrase in server plugin helper based plugins, e.g. <code>in_forward</code>/<code>out_forward</code>. </p> <h3>Major bug fixes</h3> <ul> <li>out_forward: Fix ack handling</li> <li>config: Mask all secret parameters in worker section</li> </ul> <p>This version fixed <code>out_forward</code>&#39;s <code>require_ack_response</code> regressions. If you use <code>require_ack_response</code> with v0.14, we recommend to update fluentd v0.14 to v0.14.17.</p> <p>Enjoy logging!</p> 2017-05-30 <p>Hi users!</p> <p>We have released v0.14.17. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>): This release is mainly for bug fixes.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>server: Accept private key for TLS server without passphrase</h3> <p>The <code>private_key_passphrase</code> and <code>ca_private_key_passphrase</code> for TLS setting are now optional. It means you can use private key without passphrase in server plugin helper based plugins, e.g. <code>in_forward</code>/<code>out_forward</code>. </p> <h3>Major bug fixes</h3> <ul> <li>out_forward: Fix ack handling</li> <li>config: Mask all secret parameters in worker section</li> </ul> <p>This version fixed <code>out_forward</code>&#39;s <code>require_ack_response</code> regressions. If you use <code>require_ack_response</code> with v0.14, we recommend to update fluentd v0.14 to v0.14.17.</p> <p>Enjoy logging!</p> Fluentd v0.14.16 has been released https://www.fluentd.org/blog/fluentd-v0.14.16-has-been-released <p>Hi users!</p> <p>We have released v0.14.16. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>): This release is mainly for fixing regressions.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>Allow null character in double-quoted string</h3> <p>You can use null character <code>\0</code> with double-quoted string. This is useful for delimiter, e.g. tcp input, tsv parser, etc. Here is an example:</p> <pre><code>&lt;source&gt; @type tail path /path/to/app.log tag tail.test format tsv keys a,b,c delimiter &quot;\0&quot; # \0 or &#39;\0&#39; doesn&#39;t work &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>out_forward: Call proper method for each connection type</li> <li>in_monitor_agent: check variable buffer is a Buffer instance</li> </ul> <p>Enjoy logging!</p> 2017-05-14 <p>Hi users!</p> <p>We have released v0.14.16. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>): This release is mainly for fixing regressions.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>Allow null character in double-quoted string</h3> <p>You can use null character <code>\0</code> with double-quoted string. This is useful for delimiter, e.g. tcp input, tsv parser, etc. Here is an example:</p> <pre><code>&lt;source&gt; @type tail path /path/to/app.log tag tail.test format tsv keys a,b,c delimiter &quot;\0&quot; # \0 or &#39;\0&#39; doesn&#39;t work &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>out_forward: Call proper method for each connection type</li> <li>in_monitor_agent: check variable buffer is a Buffer instance</li> </ul> <p>Enjoy logging!</p> Fluentd v0.14.15 has been released https://www.fluentd.org/blog/fluentd-v0.14.15-has-been-released <p>Hi users!</p> <p>We have released v0.14.15. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3><code>&lt;worker N&gt;</code> directive</h3> <p>We introduced built-in multi-process feature since v0.14.12(See <a href="http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released">release announcement</a>). New <code>&lt;worker N&gt;</code> directive enhances multi-process feature.</p> <p>The limitation of multi-process was non-multi-process ready plugin can&#39;t be used together. For example, we can&#39;t use in_tail plugin with <code>workers</code> parameter. Following configuration raises an error at startup.</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type forward &lt;/source&gt; &lt;source&gt; @type tail &lt;/source&gt; </code></pre> <p><code>&lt;worker N&gt;</code> directive resolves this problem. You can invoke any plugins in specific worker(<code>N</code> is worker index) and plugins under <code>&lt;worker N&gt;</code> acts as if they work on single process environment. We can mix multi-process ready and non-multi-process ready plugins in one instance. Working example with above configuration is like below:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; # work on multi process environment. worker0 - worker3 invoke in_forward plugin &lt;source&gt; @type forward &lt;/source&gt; # work on only worker 0. worker1 - worker3 don&#39;t invoke in_tail plugin &lt;worker 0&gt; &lt;source&gt; @type tail &lt;/source&gt; &lt;/worker&gt; # &lt;worker 1&gt;, &lt;worker 2&gt; or &lt;worker 3&gt; is also ok </code></pre> <h3>Plugin: Add <code>formatted_to_msgpack_binary?</code> method to Output plugin API</h3> <p>We use <code>formatted_to_msgpack_binary</code> method to tell &quot;This plugin uses msgpack for event serialization&quot; to output core when you customize <code>format</code> method. This name is not rubyish, so we add <code>formatted_to_msgpack_binary?</code> method for better name. We recommend to use <code>formatted_to_msgpack_binary?</code> for new plugins.</p> <p>Of course, <code>formatted_to_msgpack_binary</code> method is available for existing plugins.</p> <h3>Windows: Add several options to configure Fluentd windows service information</h3> <p>By default, Fluentd uses following settings for Windows service registration:</p> <ul> <li>Service name: <code>fluentdwinsvc</code></li> <li>Display name: <code>Fluentd Windows Service</code></li> <li>Service description: <code>Fluentd is an event collector system.</code></li> </ul> <p>Since v0.14.15, you can configure these settings via command line options.</p> <ul> <li>Service name: <code>--winsvc-name NAME</code></li> <li>Display name: <code>--winsvc-display-name DISPLAY_NAME</code></li> <li>Service description: <code>--winsvc-desc DESC</code></li> </ul> <h3>Major bug fixes</h3> <ul> <li>in_http: Fix X-Forwarded-For header handling. Accpet multiple headers</li> <li>windows: Quote the file path to the Ruby bin directory when starting fluentd as a windows service</li> </ul> <p>Enjoy logging!</p> 2017-04-24 <p>Hi users!</p> <p>We have released v0.14.15. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3><code>&lt;worker N&gt;</code> directive</h3> <p>We introduced built-in multi-process feature since v0.14.12(See <a href="http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released">release announcement</a>). New <code>&lt;worker N&gt;</code> directive enhances multi-process feature.</p> <p>The limitation of multi-process was non-multi-process ready plugin can&#39;t be used together. For example, we can&#39;t use in_tail plugin with <code>workers</code> parameter. Following configuration raises an error at startup.</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type forward &lt;/source&gt; &lt;source&gt; @type tail &lt;/source&gt; </code></pre> <p><code>&lt;worker N&gt;</code> directive resolves this problem. You can invoke any plugins in specific worker(<code>N</code> is worker index) and plugins under <code>&lt;worker N&gt;</code> acts as if they work on single process environment. We can mix multi-process ready and non-multi-process ready plugins in one instance. Working example with above configuration is like below:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; # work on multi process environment. worker0 - worker3 invoke in_forward plugin &lt;source&gt; @type forward &lt;/source&gt; # work on only worker 0. worker1 - worker3 don&#39;t invoke in_tail plugin &lt;worker 0&gt; &lt;source&gt; @type tail &lt;/source&gt; &lt;/worker&gt; # &lt;worker 1&gt;, &lt;worker 2&gt; or &lt;worker 3&gt; is also ok </code></pre> <h3>Plugin: Add <code>formatted_to_msgpack_binary?</code> method to Output plugin API</h3> <p>We use <code>formatted_to_msgpack_binary</code> method to tell &quot;This plugin uses msgpack for event serialization&quot; to output core when you customize <code>format</code> method. This name is not rubyish, so we add <code>formatted_to_msgpack_binary?</code> method for better name. We recommend to use <code>formatted_to_msgpack_binary?</code> for new plugins.</p> <p>Of course, <code>formatted_to_msgpack_binary</code> method is available for existing plugins.</p> <h3>Windows: Add several options to configure Fluentd windows service information</h3> <p>By default, Fluentd uses following settings for Windows service registration:</p> <ul> <li>Service name: <code>fluentdwinsvc</code></li> <li>Display name: <code>Fluentd Windows Service</code></li> <li>Service description: <code>Fluentd is an event collector system.</code></li> </ul> <p>Since v0.14.15, you can configure these settings via command line options.</p> <ul> <li>Service name: <code>--winsvc-name NAME</code></li> <li>Display name: <code>--winsvc-display-name DISPLAY_NAME</code></li> <li>Service description: <code>--winsvc-desc DESC</code></li> </ul> <h3>Major bug fixes</h3> <ul> <li>in_http: Fix X-Forwarded-For header handling. Accpet multiple headers</li> <li>windows: Quote the file path to the Ruby bin directory when starting fluentd as a windows service</li> </ul> <p>Enjoy logging!</p>