Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v0.14.16 has been released https://www.fluentd.org/blog/fluentd-v0.14.16-has-been-released <p>Hi users!</p> <p>We have released v0.14.16. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>): This release is mainly for fixing regressions.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>Allow null character in double-quoted string</h3> <p>You can use null character <code>\0</code> with double-quoted string. This is useful for delimiter, e.g. tcp input, tsv parser, etc. Here is an example:</p> <pre><code>&lt;source&gt; @type tail path /path/to/app.log tag tail.test format tsv keys a,b,c delimiter &quot;\0&quot; # \0 or &#39;\0&#39; doesn&#39;t work &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>out_forward: Call proper method for each connection type</li> <li>in_monitor_agent: check variable buffer is a Buffer instance</li> </ul> <p>Enjoy logging!</p> 2017-05-14 <p>Hi users!</p> <p>We have released v0.14.16. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>): This release is mainly for fixing regressions.</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.</p> <h3>Allow null character in double-quoted string</h3> <p>You can use null character <code>\0</code> with double-quoted string. This is useful for delimiter, e.g. tcp input, tsv parser, etc. Here is an example:</p> <pre><code>&lt;source&gt; @type tail path /path/to/app.log tag tail.test format tsv keys a,b,c delimiter &quot;\0&quot; # \0 or &#39;\0&#39; doesn&#39;t work &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>out_forward: Call proper method for each connection type</li> <li>in_monitor_agent: check variable buffer is a Buffer instance</li> </ul> <p>Enjoy logging!</p> Fluentd v0.14.15 has been released https://www.fluentd.org/blog/fluentd-v0.14.15-has-been-released <p>Hi users!</p> <p>We have released v0.14.15. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3><code>&lt;worker N&gt;</code> directive</h3> <p>We introduced built-in multi-process feature since v0.14.12(See <a href="http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released">release announcement</a>). New <code>&lt;worker N&gt;</code> directive enhances multi-process feature.</p> <p>The limitation of multi-process was non-multi-process ready plugin can&#39;t be used together. For example, we can&#39;t use in_tail plugin with <code>workers</code> parameter. Following configuration raises an error at startup.</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type forward &lt;/source&gt; &lt;source&gt; @type tail &lt;/source&gt; </code></pre> <p><code>&lt;worker N&gt;</code> directive resolves this problem. You can invoke any plugins in specific worker(<code>N</code> is worker index) and plugins under <code>&lt;worker N&gt;</code> acts as if they work on single process environment. We can mix multi-process ready and non-multi-process ready plugins in one instance. Working example with above configuration is like below:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; # work on multi process environment. worker0 - worker3 invoke in_forward plugin &lt;source&gt; @type forward &lt;/source&gt; # work on only worker 0. worker1 - worker3 don&#39;t invoke in_tail plugin &lt;worker 0&gt; &lt;source&gt; @type tail &lt;/source&gt; &lt;/worker&gt; # &lt;worker 1&gt;, &lt;worker 2&gt; or &lt;worker 3&gt; is also ok </code></pre> <h3>Plugin: Add <code>formatted_to_msgpack_binary?</code> method to Output plugin API</h3> <p>We use <code>formatted_to_msgpack_binary</code> method to tell &quot;This plugin uses msgpack for event serialization&quot; to output core when you customize <code>format</code> method. This name is not rubyish, so we add <code>formatted_to_msgpack_binary?</code> method for better name. We recommend to use <code>formatted_to_msgpack_binary?</code> for new plugins.</p> <p>Of course, <code>formatted_to_msgpack_binary</code> method is available for existing plugins.</p> <h3>Windows: Add several options to configure Fluentd windows service information</h3> <p>By default, Fluentd uses following settings for Windows service registration:</p> <ul> <li>Service name: <code>fluentdwinsvc</code></li> <li>Display name: <code>Fluentd Windows Service</code></li> <li>Service description: <code>Fluentd is an event collector system.</code></li> </ul> <p>Since v0.14.15, you can configure these settings via command line options.</p> <ul> <li>Service name: <code>--winsvc-name NAME</code></li> <li>Display name: <code>--winsvc-display-name DISPLAY_NAME</code></li> <li>Service description: <code>--winsvc-desc DESC</code></li> </ul> <h3>Major bug fixes</h3> <ul> <li>in_http: Fix X-Forwarded-For header handling. Accpet multiple headers</li> <li>windows: Quote the file path to the Ruby bin directory when starting fluentd as a windows service</li> </ul> <p>Enjoy logging!</p> 2017-04-24 <p>Hi users!</p> <p>We have released v0.14.15. Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3><code>&lt;worker N&gt;</code> directive</h3> <p>We introduced built-in multi-process feature since v0.14.12(See <a href="http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released">release announcement</a>). New <code>&lt;worker N&gt;</code> directive enhances multi-process feature.</p> <p>The limitation of multi-process was non-multi-process ready plugin can&#39;t be used together. For example, we can&#39;t use in_tail plugin with <code>workers</code> parameter. Following configuration raises an error at startup.</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type forward &lt;/source&gt; &lt;source&gt; @type tail &lt;/source&gt; </code></pre> <p><code>&lt;worker N&gt;</code> directive resolves this problem. You can invoke any plugins in specific worker(<code>N</code> is worker index) and plugins under <code>&lt;worker N&gt;</code> acts as if they work on single process environment. We can mix multi-process ready and non-multi-process ready plugins in one instance. Working example with above configuration is like below:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; # work on multi process environment. worker0 - worker3 invoke in_forward plugin &lt;source&gt; @type forward &lt;/source&gt; # work on only worker 0. worker1 - worker3 don&#39;t invoke in_tail plugin &lt;worker 0&gt; &lt;source&gt; @type tail &lt;/source&gt; &lt;/worker&gt; # &lt;worker 1&gt;, &lt;worker 2&gt; or &lt;worker 3&gt; is also ok </code></pre> <h3>Plugin: Add <code>formatted_to_msgpack_binary?</code> method to Output plugin API</h3> <p>We use <code>formatted_to_msgpack_binary</code> method to tell &quot;This plugin uses msgpack for event serialization&quot; to output core when you customize <code>format</code> method. This name is not rubyish, so we add <code>formatted_to_msgpack_binary?</code> method for better name. We recommend to use <code>formatted_to_msgpack_binary?</code> for new plugins.</p> <p>Of course, <code>formatted_to_msgpack_binary</code> method is available for existing plugins.</p> <h3>Windows: Add several options to configure Fluentd windows service information</h3> <p>By default, Fluentd uses following settings for Windows service registration:</p> <ul> <li>Service name: <code>fluentdwinsvc</code></li> <li>Display name: <code>Fluentd Windows Service</code></li> <li>Service description: <code>Fluentd is an event collector system.</code></li> </ul> <p>Since v0.14.15, you can configure these settings via command line options.</p> <ul> <li>Service name: <code>--winsvc-name NAME</code></li> <li>Display name: <code>--winsvc-display-name DISPLAY_NAME</code></li> <li>Service description: <code>--winsvc-desc DESC</code></li> </ul> <h3>Major bug fixes</h3> <ul> <li>in_http: Fix X-Forwarded-For header handling. Accpet multiple headers</li> <li>windows: Quote the file path to the Ruby bin directory when starting fluentd as a windows service</li> </ul> <p>Enjoy logging!</p> Fluentd v0.14.14 has been released https://www.fluentd.org/blog/fluentd-v0.14.14-has-been-released <p>Hi users!</p> <p>We have released v0.14.14 with several bug fixes and plugin improvement.</p> <p>Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>in_monitor_agent: Start one HTTP server per worker on sequential port numbers</h3> <p><code>in_monitor_agent</code> now launches HTTP server per worker on multiprocess environment. The port number is based on <code>port</code> parameter and worker id. For example, if you have following configuration:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type monitor_agent port 25000 &lt;/source&gt; </code></pre> <p><code>worker 1</code>&#39;s port is 25000, <code>worker 2</code>&#39;s port is 25001 and <code>worker 4</code>&#39;s port is 25003.</p> <h3>in_http: Support <code>application/msgpack</code> header</h3> <p><a href="http://fluentbit.io/">fluent-bit</a> has <code>out_http</code> and it uses <code>application/json</code> or <code>application/msgpack</code> header in the request. fluentd&#39;s <code>in_http</code> now accepts <code>application/msgpack</code> request from fluent-bit :)</p> <h3>filter_parser: Add <code>emit_invalid_record_to_error</code> parameter</h3> <p>By default, <code>filter_parser</code> emits invalid record to <code>@ERROR</code> label when parsing failed. This is useful for rescuing invalid records in your data pipeline. But users sometimes want to ignore such records for mixed logs. You can now ignore unparseable records by setting <code>emit_invalid_record_to_error false</code> in the configuration.</p> <pre><code>&lt;filter pattern&gt; @type parser # ignore non-json logs emit_invalid_record_to_error false &lt;parse&gt; @format json &lt;/parse&gt; &lt;/filter&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>config: Set encoding forcefully to avoid UndefinedConversionError</li> <li>config: Fix <code>log_level</code> handling in <system></li> <li>Fix Input and Output deadlock when buffer is full during startup</li> </ul> <p>Enjoy logging!</p> 2017-03-24 <p>Hi users!</p> <p>We have released v0.14.14 with several bug fixes and plugin improvement.</p> <p>Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>in_monitor_agent: Start one HTTP server per worker on sequential port numbers</h3> <p><code>in_monitor_agent</code> now launches HTTP server per worker on multiprocess environment. The port number is based on <code>port</code> parameter and worker id. For example, if you have following configuration:</p> <pre><code>&lt;system&gt; workers 4 &lt;system&gt; &lt;source&gt; @type monitor_agent port 25000 &lt;/source&gt; </code></pre> <p><code>worker 1</code>&#39;s port is 25000, <code>worker 2</code>&#39;s port is 25001 and <code>worker 4</code>&#39;s port is 25003.</p> <h3>in_http: Support <code>application/msgpack</code> header</h3> <p><a href="http://fluentbit.io/">fluent-bit</a> has <code>out_http</code> and it uses <code>application/json</code> or <code>application/msgpack</code> header in the request. fluentd&#39;s <code>in_http</code> now accepts <code>application/msgpack</code> request from fluent-bit :)</p> <h3>filter_parser: Add <code>emit_invalid_record_to_error</code> parameter</h3> <p>By default, <code>filter_parser</code> emits invalid record to <code>@ERROR</code> label when parsing failed. This is useful for rescuing invalid records in your data pipeline. But users sometimes want to ignore such records for mixed logs. You can now ignore unparseable records by setting <code>emit_invalid_record_to_error false</code> in the configuration.</p> <pre><code>&lt;filter pattern&gt; @type parser # ignore non-json logs emit_invalid_record_to_error false &lt;parse&gt; @format json &lt;/parse&gt; &lt;/filter&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>config: Set encoding forcefully to avoid UndefinedConversionError</li> <li>config: Fix <code>log_level</code> handling in <system></li> <li>Fix Input and Output deadlock when buffer is full during startup</li> </ul> <p>Enjoy logging!</p> Fluentd v0.12.33 has been released https://www.fluentd.org/blog/fluentd-v0.12.33-has-been-released <p>Hi users!</p> <p>We have released Fluentd version 0.12.33. Here are the changes:</p> <h3>New features / Enhancement</h3> <ul> <li>parser_syslog: Backport &#39;Support rfc5424 syslog format&#39; <a href="https://github.com/fluent/fluentd/pull/1495">https://github.com/fluent/fluentd/pull/1495</a></li> <li>in_forward: Add <code>source_address_key</code> and fix <code>source_hostname_key</code> parameters <a href="https://github.com/fluent/fluentd/pull/1490">https://github.com/fluent/fluentd/pull/1490</a></li> <li>in_tail: Add <code>limit_recently_modified</code> parameter to limit watching files <a href="https://github.com/fluent/fluentd/pull/1474">https://github.com/fluent/fluentd/pull/1474</a></li> <li>in_tail: Skip the refresh of watching list on startup <a href="https://github.com/fluent/fluentd/pull/1487">https://github.com/fluent/fluentd/pull/1487</a></li> <li>parser: Allow escape sequence in Apache access log <a href="https://github.com/fluent/fluentd/pull/1479">https://github.com/fluent/fluentd/pull/1479</a></li> <li>log: Add <code>Fluent::Log#&lt;&lt;</code> to support some SDKs <a href="https://github.com/fluent/fluentd/pull/1478">https://github.com/fluent/fluentd/pull/1478</a></li> </ul> <h3>Bug fixes</h3> <ul> <li>config: Set encoding forcefully to avoid UndefinedConversionError <a href="https://github.com/fluent/fluentd/pull/1477">https://github.com/fluent/fluentd/pull/1477</a></li> <li>in_tail: Skip directories when use<code>**/*</code> in path <a href="https://github.com/fluent/fluentd/pull/1464">https://github.com/fluent/fluentd/pull/1464</a></li> </ul> <h2>parser_syslog: Support RFC5424 format</h2> <p><code>parser_syslog</code> now supports RFC5424 format. Here is the result of RFC5424 format message:</p> <pre><code># Incoming message &lt;16&gt;1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut=&quot;3&quot; eventSource=&quot;Application&quot; eventID=&quot;11211&quot;] Hi, from Fluentd! # Parsed result {&quot;pri&quot; =&gt; 16, &quot;host&quot; =&gt; &quot;192.168.0.1&quot;, &quot;ident&quot; =&gt; &quot;fluentd&quot;, &quot;pid&quot; =&gt; &quot;11111&quot;, &quot;msgid&quot; =&gt; &quot;ID24224&quot;, &quot;extradata&quot; =&gt; &quot;[exampleSDID@20224 iut=\&quot;3\&quot; eventSource=\&quot;Application\&quot; eventID=\&quot;11211\&quot;]&quot;, &quot;message&quot; =&gt; &quot;Hi, from Fluentd!&quot;} </code></pre> <p>By default, <code>parser_syslog</code> assumes incoming message is RFC3614 format. You can parse RFC5424 format by setting <code>message_format rfc5424</code> in parser configuration. <br /> In addition, you can use <code>message_format auto</code> to handle two message formats in one stream. <code>parser_syslog</code> detects message format automatically and parse it.</p> <p>Of course, you can use this feature in <code>in_syslog</code> because <code>in_syslog</code> uses <code>parser_syslog</code> internally.</p> <pre><code class="aconf">&lt;source&gt; @type syslog @id in_syslog message_format rfc5424 tag system.syslog &lt;/source&gt; </code></pre> <h2>in_tail: Add 2 features</h2> <p>We added <code>limit_recently_modified</code> parameter. This is useful when use <code>*</code> in the path. If target files are not updated within <code>limit_recently_modified</code>, such files are ignored from watching list.</p> <p>Another one is adding <code>skip_refresh_on_startup</code> parameter. By default, <code>in_tail</code> reads file content during startup when you set <code>read_from_head true</code> or after restarted. <br /> The problem is this is done by main thread, so it blocks other startup routines. This is no problem when unread lines are small. But it is huge, consuming lines takes long time. <code>skip_refresh_on_startup</code> parameter delays the watching list update so useful for above case.</p> <p><br /> Lastly, v0.12.33 docker image has also been <a href="https://hub.docker.com/r/fluent/fluentd/tags">available on Docker Hub</a>.</p> <p><br /> Happy logging!</p> 2017-03-10 <p>Hi users!</p> <p>We have released Fluentd version 0.12.33. Here are the changes:</p> <h3>New features / Enhancement</h3> <ul> <li>parser_syslog: Backport &#39;Support rfc5424 syslog format&#39; <a href="https://github.com/fluent/fluentd/pull/1495">https://github.com/fluent/fluentd/pull/1495</a></li> <li>in_forward: Add <code>source_address_key</code> and fix <code>source_hostname_key</code> parameters <a href="https://github.com/fluent/fluentd/pull/1490">https://github.com/fluent/fluentd/pull/1490</a></li> <li>in_tail: Add <code>limit_recently_modified</code> parameter to limit watching files <a href="https://github.com/fluent/fluentd/pull/1474">https://github.com/fluent/fluentd/pull/1474</a></li> <li>in_tail: Skip the refresh of watching list on startup <a href="https://github.com/fluent/fluentd/pull/1487">https://github.com/fluent/fluentd/pull/1487</a></li> <li>parser: Allow escape sequence in Apache access log <a href="https://github.com/fluent/fluentd/pull/1479">https://github.com/fluent/fluentd/pull/1479</a></li> <li>log: Add <code>Fluent::Log#&lt;&lt;</code> to support some SDKs <a href="https://github.com/fluent/fluentd/pull/1478">https://github.com/fluent/fluentd/pull/1478</a></li> </ul> <h3>Bug fixes</h3> <ul> <li>config: Set encoding forcefully to avoid UndefinedConversionError <a href="https://github.com/fluent/fluentd/pull/1477">https://github.com/fluent/fluentd/pull/1477</a></li> <li>in_tail: Skip directories when use<code>**/*</code> in path <a href="https://github.com/fluent/fluentd/pull/1464">https://github.com/fluent/fluentd/pull/1464</a></li> </ul> <h2>parser_syslog: Support RFC5424 format</h2> <p><code>parser_syslog</code> now supports RFC5424 format. Here is the result of RFC5424 format message:</p> <pre><code># Incoming message &lt;16&gt;1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut=&quot;3&quot; eventSource=&quot;Application&quot; eventID=&quot;11211&quot;] Hi, from Fluentd! # Parsed result {&quot;pri&quot; =&gt; 16, &quot;host&quot; =&gt; &quot;192.168.0.1&quot;, &quot;ident&quot; =&gt; &quot;fluentd&quot;, &quot;pid&quot; =&gt; &quot;11111&quot;, &quot;msgid&quot; =&gt; &quot;ID24224&quot;, &quot;extradata&quot; =&gt; &quot;[exampleSDID@20224 iut=\&quot;3\&quot; eventSource=\&quot;Application\&quot; eventID=\&quot;11211\&quot;]&quot;, &quot;message&quot; =&gt; &quot;Hi, from Fluentd!&quot;} </code></pre> <p>By default, <code>parser_syslog</code> assumes incoming message is RFC3614 format. You can parse RFC5424 format by setting <code>message_format rfc5424</code> in parser configuration. <br /> In addition, you can use <code>message_format auto</code> to handle two message formats in one stream. <code>parser_syslog</code> detects message format automatically and parse it.</p> <p>Of course, you can use this feature in <code>in_syslog</code> because <code>in_syslog</code> uses <code>parser_syslog</code> internally.</p> <pre><code class="aconf">&lt;source&gt; @type syslog @id in_syslog message_format rfc5424 tag system.syslog &lt;/source&gt; </code></pre> <h2>in_tail: Add 2 features</h2> <p>We added <code>limit_recently_modified</code> parameter. This is useful when use <code>*</code> in the path. If target files are not updated within <code>limit_recently_modified</code>, such files are ignored from watching list.</p> <p>Another one is adding <code>skip_refresh_on_startup</code> parameter. By default, <code>in_tail</code> reads file content during startup when you set <code>read_from_head true</code> or after restarted. <br /> The problem is this is done by main thread, so it blocks other startup routines. This is no problem when unread lines are small. But it is huge, consuming lines takes long time. <code>skip_refresh_on_startup</code> parameter delays the watching list update so useful for above case.</p> <p><br /> Lastly, v0.12.33 docker image has also been <a href="https://hub.docker.com/r/fluent/fluentd/tags">available on Docker Hub</a>.</p> <p><br /> Happy logging!</p> Fluentd v0.14.13 has been released https://www.fluentd.org/blog/fluentd-v0.14.13-has-been-released <p>Hi users!</p> <p>We have released v0.14.13 with several bug fixes and new utilities.</p> <p>Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>Add &#39;fluent-plugin-generate&#39; command</h3> <p><code>fluent-plugin-generate</code> helps you to create new plugin for v0.14. It generates skelton code and rubygems files. Here is the step for new plugin release:</p> <pre><code># Create plugin files. generate command takes two arguments, plugin type and name % fluent-plugin-generate output test License: Apache-2.0 create Gemfile create README.md create Rakefile create fluent-plugin-test.gemspec create lib/fluent/plugin/out_test.rb create test/helper.rb create test/plugin/test_out_test.rb Initialized empty Git repository in /Users/repeatedly/tmp/fluent-plugin-test/.git/ # Set up develop environment % gem install bundler # if you don&#39;t install bundler yet % edit fluent-plugin-test.gemspec # need to update summary, description and homepage to use bundle % bundle install --path vendor/bundle % bundle exec rake test # check running test # edit plugin code and test % edit lib/fluent/plugin/out_test.rb % edit test/plugin/test_out_test.rb # Update README and publish gem % edit README.md % bundle exec rake release </code></pre> <p>Currently, our plugin development article for v0.14 is incomplete. We will write new article with v0.14 API soon.</p> <h3>Skip record with Output#format API</h3> <p>In v0.12, Output plugin can skip invalid / unexpected record by using <code>Fluent::BufferedOutput#format_stream</code> API. But v0.14 API doesn&#39;t provide it so hard to migrate v0.12 plugin into v0.14.</p> <p>To avoid this problem, <code>Fluent::Plugin::Output#format</code> API can return <code>nil</code> to skip the record. Here is an example:</p> <pre><code>helpers :event_emitter # need this code to use router in Output plugin def format(tag, time, record) if record[&#39;field&#39;] == &#39;foo&#39; record.to_json + &quot;\n&quot; else router.emit_error_event(tag, time, record, ArgumentError.new(&quot;received event is invalid in xxx plugin&quot;)) nil end end </code></pre> <p>With above code, if the value of <code>field</code> is <code>&#39;foo&#39;</code>, its events are stored into buffer. On the other hand, other events are routed to error stream and sucn events are not stored in buffer.</p> <h3>Major bug fixes</h3> <ul> <li>output: Secondary calculation should consider <code>retry_max_times</code></li> <li>parser: Fix missing <code>parser_regex</code> require</li> <li>in_tail: Untracked files should be removed from watching list to avoid memory bloat</li> <li>record_transformer: Revert &quot;Use BasicObject for cleanroom&quot; for <code>enable_ruby</code> regression.</li> </ul> <p>Enjoy logging!</p> 2017-02-20 <p>Hi users!</p> <p>We have released v0.14.13 with several bug fixes and new utilities.</p> <p>Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>Add &#39;fluent-plugin-generate&#39; command</h3> <p><code>fluent-plugin-generate</code> helps you to create new plugin for v0.14. It generates skelton code and rubygems files. Here is the step for new plugin release:</p> <pre><code># Create plugin files. generate command takes two arguments, plugin type and name % fluent-plugin-generate output test License: Apache-2.0 create Gemfile create README.md create Rakefile create fluent-plugin-test.gemspec create lib/fluent/plugin/out_test.rb create test/helper.rb create test/plugin/test_out_test.rb Initialized empty Git repository in /Users/repeatedly/tmp/fluent-plugin-test/.git/ # Set up develop environment % gem install bundler # if you don&#39;t install bundler yet % edit fluent-plugin-test.gemspec # need to update summary, description and homepage to use bundle % bundle install --path vendor/bundle % bundle exec rake test # check running test # edit plugin code and test % edit lib/fluent/plugin/out_test.rb % edit test/plugin/test_out_test.rb # Update README and publish gem % edit README.md % bundle exec rake release </code></pre> <p>Currently, our plugin development article for v0.14 is incomplete. We will write new article with v0.14 API soon.</p> <h3>Skip record with Output#format API</h3> <p>In v0.12, Output plugin can skip invalid / unexpected record by using <code>Fluent::BufferedOutput#format_stream</code> API. But v0.14 API doesn&#39;t provide it so hard to migrate v0.12 plugin into v0.14.</p> <p>To avoid this problem, <code>Fluent::Plugin::Output#format</code> API can return <code>nil</code> to skip the record. Here is an example:</p> <pre><code>helpers :event_emitter # need this code to use router in Output plugin def format(tag, time, record) if record[&#39;field&#39;] == &#39;foo&#39; record.to_json + &quot;\n&quot; else router.emit_error_event(tag, time, record, ArgumentError.new(&quot;received event is invalid in xxx plugin&quot;)) nil end end </code></pre> <p>With above code, if the value of <code>field</code> is <code>&#39;foo&#39;</code>, its events are stored into buffer. On the other hand, other events are routed to error stream and sucn events are not stored in buffer.</p> <h3>Major bug fixes</h3> <ul> <li>output: Secondary calculation should consider <code>retry_max_times</code></li> <li>parser: Fix missing <code>parser_regex</code> require</li> <li>in_tail: Untracked files should be removed from watching list to avoid memory bloat</li> <li>record_transformer: Revert &quot;Use BasicObject for cleanroom&quot; for <code>enable_ruby</code> regression.</li> </ul> <p>Enjoy logging!</p>