Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v0.12.33 has been released https://www.fluentd.org/blog/fluentd-v0.12.33-has-been-released <p>Hi users!</p> <p>We have released Fluentd version 0.12.33. Here are the changes:</p> <h3>New features / Enhancement</h3> <ul> <li>parser_syslog: Backport &#39;Support rfc5424 syslog format&#39; <a href="https://github.com/fluent/fluentd/pull/1495">https://github.com/fluent/fluentd/pull/1495</a></li> <li>in_forward: Add <code>source_address_key</code> and fix <code>source_hostname_key</code> parameters <a href="https://github.com/fluent/fluentd/pull/1490">https://github.com/fluent/fluentd/pull/1490</a></li> <li>in_tail: Add <code>limit_recently_modified</code> parameter to limit watching files <a href="https://github.com/fluent/fluentd/pull/1474">https://github.com/fluent/fluentd/pull/1474</a></li> <li>in_tail: Skip the refresh of watching list on startup <a href="https://github.com/fluent/fluentd/pull/1487">https://github.com/fluent/fluentd/pull/1487</a></li> <li>parser: Allow escape sequence in Apache access log <a href="https://github.com/fluent/fluentd/pull/1479">https://github.com/fluent/fluentd/pull/1479</a></li> <li>log: Add <code>Fluent::Log#&lt;&lt;</code> to support some SDKs <a href="https://github.com/fluent/fluentd/pull/1478">https://github.com/fluent/fluentd/pull/1478</a></li> </ul> <h3>Bug fixes</h3> <ul> <li>config: Set encoding forcefully to avoid UndefinedConversionError <a href="https://github.com/fluent/fluentd/pull/1477">https://github.com/fluent/fluentd/pull/1477</a></li> <li>in_tail: Skip directories when use<code>**/*</code> in path <a href="https://github.com/fluent/fluentd/pull/1464">https://github.com/fluent/fluentd/pull/1464</a></li> </ul> <h2>parser_syslog: Support RFC5424 format</h2> <p><code>parser_syslog</code> now supports RFC5424 format. Here is the result of RFC5424 format message:</p> <pre><code># Incoming message &lt;16&gt;1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut=&quot;3&quot; eventSource=&quot;Application&quot; eventID=&quot;11211&quot;] Hi, from Fluentd! # Parsed result {&quot;pri&quot; =&gt; 16, &quot;host&quot; =&gt; &quot;192.168.0.1&quot;, &quot;ident&quot; =&gt; &quot;fluentd&quot;, &quot;pid&quot; =&gt; &quot;11111&quot;, &quot;msgid&quot; =&gt; &quot;ID24224&quot;, &quot;extradata&quot; =&gt; &quot;[exampleSDID@20224 iut=\&quot;3\&quot; eventSource=\&quot;Application\&quot; eventID=\&quot;11211\&quot;]&quot;, &quot;message&quot; =&gt; &quot;Hi, from Fluentd!&quot;} </code></pre> <p>By default, <code>parser_syslog</code> assumes incoming message is RFC3614 format. You can parse RFC5424 format by setting <code>message_format rfc5424</code> in parser configuration. <br /> In addition, you can use <code>message_format auto</code> to handle two message formats in one stream. <code>parser_syslog</code> detects message format automatically and parse it.</p> <p>Of course, you can use this feature in <code>in_syslog</code> because <code>in_syslog</code> uses <code>parser_syslog</code> internally.</p> <pre><code class="aconf">&lt;source&gt; @type syslog @id in_syslog message_format rfc5424 tag system.syslog &lt;/source&gt; </code></pre> <h2>in_tail: Add 2 features</h2> <p>We added <code>limit_recently_modified</code> parameter. This is useful when use <code>*</code> in the path. If target files are not updated within <code>limit_recently_modified</code>, such files are ignored from watching list.</p> <p>Another one is adding <code>skip_refresh_on_startup</code> parameter. By default, <code>in_tail</code> reads file content during startup when you set <code>read_from_head true</code> or after restarted. <br /> The problem is this is done by main thread, so it blocks other startup routines. This is no problem when unread lines are small. But it is huge, consuming lines takes long time. <code>skip_refresh_on_startup</code> parameter delays the watching list update so useful for above case.</p> <p><br /> Lastly, v0.12.33 docker image has also been <a href="https://hub.docker.com/r/fluent/fluentd/tags">available on Docker Hub</a>.</p> <p><br /> Happy logging!</p> 2017-03-10 <p>Hi users!</p> <p>We have released Fluentd version 0.12.33. Here are the changes:</p> <h3>New features / Enhancement</h3> <ul> <li>parser_syslog: Backport &#39;Support rfc5424 syslog format&#39; <a href="https://github.com/fluent/fluentd/pull/1495">https://github.com/fluent/fluentd/pull/1495</a></li> <li>in_forward: Add <code>source_address_key</code> and fix <code>source_hostname_key</code> parameters <a href="https://github.com/fluent/fluentd/pull/1490">https://github.com/fluent/fluentd/pull/1490</a></li> <li>in_tail: Add <code>limit_recently_modified</code> parameter to limit watching files <a href="https://github.com/fluent/fluentd/pull/1474">https://github.com/fluent/fluentd/pull/1474</a></li> <li>in_tail: Skip the refresh of watching list on startup <a href="https://github.com/fluent/fluentd/pull/1487">https://github.com/fluent/fluentd/pull/1487</a></li> <li>parser: Allow escape sequence in Apache access log <a href="https://github.com/fluent/fluentd/pull/1479">https://github.com/fluent/fluentd/pull/1479</a></li> <li>log: Add <code>Fluent::Log#&lt;&lt;</code> to support some SDKs <a href="https://github.com/fluent/fluentd/pull/1478">https://github.com/fluent/fluentd/pull/1478</a></li> </ul> <h3>Bug fixes</h3> <ul> <li>config: Set encoding forcefully to avoid UndefinedConversionError <a href="https://github.com/fluent/fluentd/pull/1477">https://github.com/fluent/fluentd/pull/1477</a></li> <li>in_tail: Skip directories when use<code>**/*</code> in path <a href="https://github.com/fluent/fluentd/pull/1464">https://github.com/fluent/fluentd/pull/1464</a></li> </ul> <h2>parser_syslog: Support RFC5424 format</h2> <p><code>parser_syslog</code> now supports RFC5424 format. Here is the result of RFC5424 format message:</p> <pre><code># Incoming message &lt;16&gt;1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut=&quot;3&quot; eventSource=&quot;Application&quot; eventID=&quot;11211&quot;] Hi, from Fluentd! # Parsed result {&quot;pri&quot; =&gt; 16, &quot;host&quot; =&gt; &quot;192.168.0.1&quot;, &quot;ident&quot; =&gt; &quot;fluentd&quot;, &quot;pid&quot; =&gt; &quot;11111&quot;, &quot;msgid&quot; =&gt; &quot;ID24224&quot;, &quot;extradata&quot; =&gt; &quot;[exampleSDID@20224 iut=\&quot;3\&quot; eventSource=\&quot;Application\&quot; eventID=\&quot;11211\&quot;]&quot;, &quot;message&quot; =&gt; &quot;Hi, from Fluentd!&quot;} </code></pre> <p>By default, <code>parser_syslog</code> assumes incoming message is RFC3614 format. You can parse RFC5424 format by setting <code>message_format rfc5424</code> in parser configuration. <br /> In addition, you can use <code>message_format auto</code> to handle two message formats in one stream. <code>parser_syslog</code> detects message format automatically and parse it.</p> <p>Of course, you can use this feature in <code>in_syslog</code> because <code>in_syslog</code> uses <code>parser_syslog</code> internally.</p> <pre><code class="aconf">&lt;source&gt; @type syslog @id in_syslog message_format rfc5424 tag system.syslog &lt;/source&gt; </code></pre> <h2>in_tail: Add 2 features</h2> <p>We added <code>limit_recently_modified</code> parameter. This is useful when use <code>*</code> in the path. If target files are not updated within <code>limit_recently_modified</code>, such files are ignored from watching list.</p> <p>Another one is adding <code>skip_refresh_on_startup</code> parameter. By default, <code>in_tail</code> reads file content during startup when you set <code>read_from_head true</code> or after restarted. <br /> The problem is this is done by main thread, so it blocks other startup routines. This is no problem when unread lines are small. But it is huge, consuming lines takes long time. <code>skip_refresh_on_startup</code> parameter delays the watching list update so useful for above case.</p> <p><br /> Lastly, v0.12.33 docker image has also been <a href="https://hub.docker.com/r/fluent/fluentd/tags">available on Docker Hub</a>.</p> <p><br /> Happy logging!</p> Fluentd v0.14.13 has been released https://www.fluentd.org/blog/fluentd-v0.14.13-has-been-released <p>Hi users!</p> <p>We have released v0.14.13 with several bug fixes and new utilities.</p> <p>Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>Add &#39;fluent-plugin-generate&#39; command</h3> <p><code>fluent-plugin-generate</code> helps you to create new plugin for v0.14. It generates skelton code and rubygems files. Here is the step for new plugin release:</p> <pre><code># Create plugin files. generate command takes two arguments, plugin type and name % fluent-plugin-generate output test License: Apache-2.0 create Gemfile create README.md create Rakefile create fluent-plugin-test.gemspec create lib/fluent/plugin/out_test.rb create test/helper.rb create test/plugin/test_out_test.rb Initialized empty Git repository in /Users/repeatedly/tmp/fluent-plugin-test/.git/ # Set up develop environment % gem install bundler # if you don&#39;t install bundler yet % edit fluent-plugin-test.gemspec # need to update summary, description and homepage to use bundle % bundle install --path vendor/bundle % bundle exec rake test # check running test # edit plugin code and test % edit lib/fluent/plugin/out_test.rb % edit test/plugin/test_out_test.rb # Update README and publish gem % edit README.md % bundle exec rake release </code></pre> <p>Currently, our plugin development article for v0.14 is incomplete. We will write new article with v0.14 API soon.</p> <h3>Skip record with Output#format API</h3> <p>In v0.12, Output plugin can skip invalid / unexpected record by using <code>Fluent::BufferedOutput#format_stream</code> API. But v0.14 API doesn&#39;t provide it so hard to migrate v0.12 plugin into v0.14.</p> <p>To avoid this problem, <code>Fluent::Plugin::Output#format</code> API can return <code>nil</code> to skip the record. Here is an example:</p> <pre><code>helpers :event_emitter # need this code to use router in Output plugin def format(tag, time, record) if record[&#39;field&#39;] == &#39;foo&#39; record.to_json + &quot;\n&quot; else router.emit_error_event(tag, time, record, ArgumentError.new(&quot;received event is invalid in xxx plugin&quot;)) nil end end </code></pre> <p>With above code, if the value of <code>field</code> is <code>&#39;foo&#39;</code>, its events are stored into buffer. On the other hand, other events are routed to error stream and sucn events are not stored in buffer.</p> <h3>Major bug fixes</h3> <ul> <li>output: Secondary calculation should consider <code>retry_max_times</code></li> <li>parser: Fix missing <code>parser_regex</code> require</li> <li>in_tail: Untracked files should be removed from watching list to avoid memory bloat</li> <li>record_transformer: Revert &quot;Use BasicObject for cleanroom&quot; for <code>enable_ruby</code> regression.</li> </ul> <p>Enjoy logging!</p> 2017-02-20 <p>Hi users!</p> <p>We have released v0.14.13 with several bug fixes and new utilities.</p> <p>Here are the changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>Add &#39;fluent-plugin-generate&#39; command</h3> <p><code>fluent-plugin-generate</code> helps you to create new plugin for v0.14. It generates skelton code and rubygems files. Here is the step for new plugin release:</p> <pre><code># Create plugin files. generate command takes two arguments, plugin type and name % fluent-plugin-generate output test License: Apache-2.0 create Gemfile create README.md create Rakefile create fluent-plugin-test.gemspec create lib/fluent/plugin/out_test.rb create test/helper.rb create test/plugin/test_out_test.rb Initialized empty Git repository in /Users/repeatedly/tmp/fluent-plugin-test/.git/ # Set up develop environment % gem install bundler # if you don&#39;t install bundler yet % edit fluent-plugin-test.gemspec # need to update summary, description and homepage to use bundle % bundle install --path vendor/bundle % bundle exec rake test # check running test # edit plugin code and test % edit lib/fluent/plugin/out_test.rb % edit test/plugin/test_out_test.rb # Update README and publish gem % edit README.md % bundle exec rake release </code></pre> <p>Currently, our plugin development article for v0.14 is incomplete. We will write new article with v0.14 API soon.</p> <h3>Skip record with Output#format API</h3> <p>In v0.12, Output plugin can skip invalid / unexpected record by using <code>Fluent::BufferedOutput#format_stream</code> API. But v0.14 API doesn&#39;t provide it so hard to migrate v0.12 plugin into v0.14.</p> <p>To avoid this problem, <code>Fluent::Plugin::Output#format</code> API can return <code>nil</code> to skip the record. Here is an example:</p> <pre><code>helpers :event_emitter # need this code to use router in Output plugin def format(tag, time, record) if record[&#39;field&#39;] == &#39;foo&#39; record.to_json + &quot;\n&quot; else router.emit_error_event(tag, time, record, ArgumentError.new(&quot;received event is invalid in xxx plugin&quot;)) nil end end </code></pre> <p>With above code, if the value of <code>field</code> is <code>&#39;foo&#39;</code>, its events are stored into buffer. On the other hand, other events are routed to error stream and sucn events are not stored in buffer.</p> <h3>Major bug fixes</h3> <ul> <li>output: Secondary calculation should consider <code>retry_max_times</code></li> <li>parser: Fix missing <code>parser_regex</code> require</li> <li>in_tail: Untracked files should be removed from watching list to avoid memory bloat</li> <li>record_transformer: Revert &quot;Use BasicObject for cleanroom&quot; for <code>enable_ruby</code> regression.</li> </ul> <p>Enjoy logging!</p> Debian based fluentd docker image has been released https://www.fluentd.org/blog/Debian-based-fluentd-docker-image <p>Hi users!</p> <p>We officially provide Alpine Linux based Fluentd docker image. Alpine Linux is light weight and this is good for Fluentd use cases. But some plugins, e.g. <code>fluent-plugin-systemd</code>, don&#39;t work on Alpine Linux, so we received the request &quot;Could you provide other OS based image for xxx.&quot;</p> <p>To resolve this problem, <a href="https://github.com/tyranron">@tyranron</a> works on improving Docker image management. Easy to add other OS, more automated release and tagging, build testing and more. Thanks to @tyranron for your hard work :)</p> <p>In this result, we start to provide Debian based docker image together.</p> <p><a href="https://github.com/fluent/fluentd-docker-image/pull/71">Debian images by tyranron · Pull Request #71 · fluent/fluentd-docker-image</a></p> <p>The default is still Alpine Linux but you can choose Debian version for your requirement. See <a href="https://github.com/fluent/fluentd-docker-image#supported-tags-and-respective-dockerfile-links">Supported tags</a> section in README for all available images.</p> <p>If you have any problem, please let me know.</p> <p><br /> Happy logging!</p> 2017-02-09 <p>Hi users!</p> <p>We officially provide Alpine Linux based Fluentd docker image. Alpine Linux is light weight and this is good for Fluentd use cases. But some plugins, e.g. <code>fluent-plugin-systemd</code>, don&#39;t work on Alpine Linux, so we received the request &quot;Could you provide other OS based image for xxx.&quot;</p> <p>To resolve this problem, <a href="https://github.com/tyranron">@tyranron</a> works on improving Docker image management. Easy to add other OS, more automated release and tagging, build testing and more. Thanks to @tyranron for your hard work :)</p> <p>In this result, we start to provide Debian based docker image together.</p> <p><a href="https://github.com/fluent/fluentd-docker-image/pull/71">Debian images by tyranron · Pull Request #71 · fluent/fluentd-docker-image</a></p> <p>The default is still Alpine Linux but you can choose Debian version for your requirement. See <a href="https://github.com/fluent/fluentd-docker-image#supported-tags-and-respective-dockerfile-links">Supported tags</a> section in README for all available images.</p> <p>If you have any problem, please let me know.</p> <p><br /> Happy logging!</p> Fluentd v0.12.32 has been released https://www.fluentd.org/blog/fluentd-v0.12.32-has-been-released <p>Hi users!</p> <p>We have released Fluentd version 0.12.32. Here are the changes:</p> <h3>New features / Enhancement</h3> <ul> <li>formatter: Add <code>add_newline</code> parameter to remove <code>\n</code> from the result <a href="https://github.com/fluent/fluentd/pull/1447">https://github.com/fluent/fluentd/pull/1447</a></li> <li>in_tail: Capture unmatched lines <a href="https://github.com/fluent/fluentd/pull/1421">https://github.com/fluent/fluentd/pull/1421</a></li> <li>in_monitor_agent: enable to select instance variables of plugins <a href="https://github.com/fluent/fluentd/pull/1402">https://github.com/fluent/fluentd/pull/1402</a></li> <li>in_monitor_agent: Port #1387 to v0.12 for retry field <a href="https://github.com/fluent/fluentd/pull/1387">https://github.com/fluent/fluentd/pull/1387</a></li> <li>Allow msgpack-ruby v1</li> </ul> <h3>Bug fixes</h3> <ul> <li>in_exec: Ensure to sleep for prevending fork bomb <a href="https://github.com/fluent/fluentd/pull/1348">https://github.com/fluent/fluentd/pull/1348</a></li> <li>record_transformer: <code>remove_keys</code> processing should be last <a href="https://github.com/fluent/fluentd/pull/1433">https://github.com/fluent/fluentd/pull/1433</a></li> </ul> <h2>formatter: Add add_newline parameter to remove &#39;\n&#39; from the result</h2> <p><code>json</code>, <code>ltsv</code>, <code>csv</code> and <code>hash</code> formatters now support <code>add_newline</code> parameter. If you set <code>add_newline false</code>, the formatted result doesn&#39;t contain <code>\n</code>. This is useful for Queue / KVS plugins.</p> <h2>in_tail: Capture unmatched lines</h2> <p>v0.12.32 introduces <code>emit_unmatched_lines</code> parameter to <code>in_tail</code> plugin. This option enables you to get unparseable line as an event.</p> <p>For example, if you have following configuration and file:</p> <ul> <li>fluent.conf</li> </ul> <pre><code>&lt;source&gt; @type tail path /path/to/json.log format json emit_unmatched_lines true # ... &lt;/source&gt; </code></pre> <ul> <li>json.log</li> </ul> <pre><code>{&quot;message&quot;:&quot;Hello!&quot;} This is broken line! {&quot;message&quot;:&quot;Yay :)&quot;} </code></pre> <p>Then, the following events are generated from this <code>tail</code> plugin.</p> <pre><code>{&quot;message&quot;:&quot;Hello!&quot;} {&quot;unmatched_line&quot;:&quot;This is broken line!&quot;} {&quot; message&quot;:&quot;Yay :)&quot;} </code></pre> <p>Unmatched line is assigned to <code>unmatched_line</code> field.</p> <p>This option works with both single-line and multi-lines.</p> <h2>in_monitor_agent: Add <code>retry</code> field</h2> <p>Fluentd v0.14 adds <code>retry</code> field to <code>/api/plugins.json</code> response. The <code>retry</code> field contains detailed information of buffer&#39;s retry. For API consistency, v0.12&#39;s <code>in_monitor_agent</code> also provides same field. Here is <code>retry</code> field example:</p> <pre><code>{ // other monitor_agent keys &quot;retry&quot;: { &quot;steps&quot;:1, // same as retry_count &quot;next_time&quot;:&quot;2016-12-23 01:37:16 +0700&quot; } } </code></pre> <p>Unlike v0.14, <code>start</code> field doesn&#39;t exist because v0.12&#39;s buffer doesn&#39;t have its information.</p> <p>We recommend to use <code>retry</code>&#39;s <code>steps</code> instead of <code>retry_count</code> for monitoring.</p> <p><br /> Lastly, v0.12.32 docker image has also been <a href="https://hub.docker.com/r/fluent/fluentd/tags">available on Docker Hub</a>.</p> <p><br /> Happy logging!</p> 2017-02-03 <p>Hi users!</p> <p>We have released Fluentd version 0.12.32. Here are the changes:</p> <h3>New features / Enhancement</h3> <ul> <li>formatter: Add <code>add_newline</code> parameter to remove <code>\n</code> from the result <a href="https://github.com/fluent/fluentd/pull/1447">https://github.com/fluent/fluentd/pull/1447</a></li> <li>in_tail: Capture unmatched lines <a href="https://github.com/fluent/fluentd/pull/1421">https://github.com/fluent/fluentd/pull/1421</a></li> <li>in_monitor_agent: enable to select instance variables of plugins <a href="https://github.com/fluent/fluentd/pull/1402">https://github.com/fluent/fluentd/pull/1402</a></li> <li>in_monitor_agent: Port #1387 to v0.12 for retry field <a href="https://github.com/fluent/fluentd/pull/1387">https://github.com/fluent/fluentd/pull/1387</a></li> <li>Allow msgpack-ruby v1</li> </ul> <h3>Bug fixes</h3> <ul> <li>in_exec: Ensure to sleep for prevending fork bomb <a href="https://github.com/fluent/fluentd/pull/1348">https://github.com/fluent/fluentd/pull/1348</a></li> <li>record_transformer: <code>remove_keys</code> processing should be last <a href="https://github.com/fluent/fluentd/pull/1433">https://github.com/fluent/fluentd/pull/1433</a></li> </ul> <h2>formatter: Add add_newline parameter to remove &#39;\n&#39; from the result</h2> <p><code>json</code>, <code>ltsv</code>, <code>csv</code> and <code>hash</code> formatters now support <code>add_newline</code> parameter. If you set <code>add_newline false</code>, the formatted result doesn&#39;t contain <code>\n</code>. This is useful for Queue / KVS plugins.</p> <h2>in_tail: Capture unmatched lines</h2> <p>v0.12.32 introduces <code>emit_unmatched_lines</code> parameter to <code>in_tail</code> plugin. This option enables you to get unparseable line as an event.</p> <p>For example, if you have following configuration and file:</p> <ul> <li>fluent.conf</li> </ul> <pre><code>&lt;source&gt; @type tail path /path/to/json.log format json emit_unmatched_lines true # ... &lt;/source&gt; </code></pre> <ul> <li>json.log</li> </ul> <pre><code>{&quot;message&quot;:&quot;Hello!&quot;} This is broken line! {&quot;message&quot;:&quot;Yay :)&quot;} </code></pre> <p>Then, the following events are generated from this <code>tail</code> plugin.</p> <pre><code>{&quot;message&quot;:&quot;Hello!&quot;} {&quot;unmatched_line&quot;:&quot;This is broken line!&quot;} {&quot; message&quot;:&quot;Yay :)&quot;} </code></pre> <p>Unmatched line is assigned to <code>unmatched_line</code> field.</p> <p>This option works with both single-line and multi-lines.</p> <h2>in_monitor_agent: Add <code>retry</code> field</h2> <p>Fluentd v0.14 adds <code>retry</code> field to <code>/api/plugins.json</code> response. The <code>retry</code> field contains detailed information of buffer&#39;s retry. For API consistency, v0.12&#39;s <code>in_monitor_agent</code> also provides same field. Here is <code>retry</code> field example:</p> <pre><code>{ // other monitor_agent keys &quot;retry&quot;: { &quot;steps&quot;:1, // same as retry_count &quot;next_time&quot;:&quot;2016-12-23 01:37:16 +0700&quot; } } </code></pre> <p>Unlike v0.14, <code>start</code> field doesn&#39;t exist because v0.12&#39;s buffer doesn&#39;t have its information.</p> <p>We recommend to use <code>retry</code>&#39;s <code>steps</code> instead of <code>retry_count</code> for monitoring.</p> <p><br /> Lastly, v0.12.32 docker image has also been <a href="https://hub.docker.com/r/fluent/fluentd/tags">available on Docker Hub</a>.</p> <p><br /> Happy logging!</p> Fluentd v0.14.12 has been released https://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released <p>Hi users!</p> <p>After the release of Fluentd v0.14.10, we did 2 releases, v0.14.11 at the end of 2016, and v0.14.12 today. Fluentd v0.14.11 release was a kind of quick fix for major bug. And now, we&#39;re very happy to introduce three major new feature with Fluentd v0.14.12!</p> <p>Here are major changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. Newly added features are under experimental... we need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>Root directory, and no more &quot;path&quot; parameters in buffer configuration</h3> <p>With past versions of Fluentd, file buffer plugin requires <code>path</code> parameter to store buffer chunks in local file systems. These paths should be configured not to use same directories carefully.</p> <pre><code>&lt;match log.service&gt; @id forward_a @type forward &lt;buffer&gt; @type file path /path/of/buffer/for/service flush_interval 1s &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; &lt;match log.system&gt; @id forward_b @type forward &lt;buffer&gt; @type file path /path/of/buffer/for/systems flush_interval 1s &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; </code></pre> <p>With Fluentd v0.14.12, these paths can be configured automatically, using <code>root_dir</code> option in <code>&lt;system&gt;</code> directive.</p> <pre><code>&lt;system&gt; root_dir /path/fluentd/root &lt;/system&gt; &lt;label @traffic&gt; &lt;match log.service&gt; @id forward_a @type forward &lt;buffer&gt; @type file flush_interval 1s &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; &lt;match log.system&gt; @id forward_b @type forward &lt;buffer&gt; @type file flush_interval 1s &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; &lt;/label&gt; </code></pre> <p>There are no need to configure <code>path</code> in <code>&lt;buffer&gt;</code> sections when <code>root_dir</code> is configured and all plugins are configured with <code>@id</code> parameter. All buffer chunks (and storage plugin files) will be stored under <code>/path/fluentd/root/worker0/plugin_id</code>. Don&#39;t miss to monitor the disk usage of that partition!</p> <h3>Multi process workers</h3> <p>Fluentd v0.14.12 supports multi process workers, finally! With this feature, two or more Fluentd workers will be launched, using the same number of CPU cores. This feature can replace <a href="https://github.com/fluent/fluent-plugin-multiprocess">fluent-plugin-multiprocess</a>, with very simple configuration, to process huge traffic on multi CPU servers.</p> <p>Note: There are a requirement: <strong>All configured plugin must support multi process workers feature</strong>.</p> <pre><code>&lt;system&gt; workers 8 # also can be specified by `--workers` command line option root_dir /path/fluentd/root # strongly recommended to specify this &lt;/system&gt; &lt;source&gt; @id input_from_edge @type forward @label @aggr_traffic port 24224 &lt;/source&gt; &lt;label @aggr_traffic&gt; &lt;match **&gt; @id output_to_backend @type forward &lt;buffer&gt; @type file flush_interval 1s chunk_limit_size 8m total_limit_size 64g # limit per plugin instance, per worker (== 64GB * 8 in total max) &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; &lt;/label&gt; </code></pre> <p>Eight worker (and a supervisor) processes work like below:</p> <ul> <li>Fluentd launches 8 worker processes in the startup sequence</li> <li>Supervisor process listens <code>port 24224</code></li> <li>When TCP connection requests arrived from client, supervisor delegates <code>accept</code> to a worker of workers (server plugin helper does it)</li> <li>A worker accepts the connection and process inbound traffic from that socket</li> </ul> <p>All worker processes have the set of plugins, routing and buffers (under <code>root_dir/workerN</code> if <code>root_dir</code> is configured), and work in parallel completely. Each workers consumes memory and disk spaces - you should take care to configure buffer spaces, and to monitor memory/disk consumption.</p> <p>This feature is highly experimental. We need many feedbacks, including bug reports, performance graphs, resource consumption summary and more.</p> <h4>Making plugins available with multi process workers</h4> <p>Once again, multi process workers configuration requires all configured plugins support multi process worker feature. It means:</p> <ul> <li>Plugin MUST be implemented with v0.14 API natively <ul> <li>Plugin class is a direct subclass of <code>Fluent::Plugin::Input</code>, <code>Fluent::Plugin::Output</code>, <code>Fluent::Plugin::Filters</code>, etc</li> </ul></li> <li>Plugin MUST have <code>#multi_workers_ready?</code> method and return <code>true</code> <ul> <li>The default return value of this method is <code>true</code> in filter/buffer/parser/formatter/storage plugins</li> <li>But it&#39;s <code>false</code> in input/output plugins</li> <li>3rd party plugins SHOULD re-implemnet <code>#multi_workers_ready?</code> to return <code>true</code> to support multi process workers</li> </ul></li> <li>Plugin MUST NOT use raw sockets to listen network ports <ul> <li>It makes <code>Errno::ADDRINUSE</code> errors with multi process worker configurations</li> <li>Use server plugin helper to create networks servers to listen network port via the supervisor process</li> </ul></li> </ul> <p>Almost all built-in plugins support multi process workers, but <code>in_tail</code> doesn&#39;t support it. If <code>in_tail</code> is configured with <code>workers</code>, Fluentd will fail to start. This disadvantage will be solved in future versions (<a href="https://github.com/fluent/fluentd/issues/1392">see also</a>).</p> <h3>TLS encrypted communication support</h3> <p>There has been many requests to support secure network transportation between Fluentd nodes, for the cases of communication between data centers. Some users had used 3rd party <a href="https://github.com/tagomoris/fluent-plugin-secure-forward">fluent-plugin-secure-forward</a> plugin... But finally, Fluentd core supports it!</p> <p>Here&#39;s a configuration example to use server certificates for server nodes (input side), and to share it in client side (output side) too:</p> <pre><code>##### output side &lt;match secret.log&gt; @id datacenter_a_output @type forward transport tls tls_cert_path /etc/mysecret/cert.pem # server certification file of destination node # or private CA, or ... ### tls_cert_path can be specified with 2 or more paths # tls_cert_path /etc/mysecret/cert1.pem, /etc/mysecret/cert2.pem ### There are some options for detailed configuration.... # tls_version TLSv1_2 # TLS v1.2 (or TLSv1_1) # tls_insecure_mode false # skip all certificate checks if true # tls_allow_self_signed_cert false # tls_verify_hostname true # verify FQDN in certificates, with SNI &lt;server&gt; host dest1.myhost.example.com port 24228 &lt;/server&gt; &lt;server&gt; # or, use host(ip address) &amp; name(hostname to verify with cert) host 203.0.113.103 name dest2.myhost.example.com port 24228 &lt;/server&gt; &lt;/match&gt; ##### input side &lt;source&gt; @id datacenter_b_input @type forward port 24228 &lt;transport tls&gt; # version TLSv1_2 cert_path /etc/mysecret/cert1 private_key_path /etc/mysecret/server.key.pem private_key_passphrase my_secret_passphrase_for_key &lt;/transport&gt; &lt;/source&gt; </code></pre> <p>If your server certificate requires intermediate CA certificates to be verified, concat these certs to make chained certs (just like nginx) and specify it in <code>cert_path</code>.</p> <p>There are some patterns to configure <code>forward</code> input plugin with TLS. All parameters for input plugin (server side) below should be specified in <code>&lt;transport tls&gt;</code> section.</p> <ul> <li>Server certificates signed by public CA <ul> <li>specify <code>cert_path</code>, <code>private_key_path</code> and <code>private_key_passphrase</code> in server side</li> <li>just specify <code>transport tls</code> in clien side (if your system&#39;s cert store has valid root CA certs)</li> </ul></li> <li>Server certificates signed by private CA <ul> <li>specify <code>cert_path</code>, <code>private_key_path</code> and <code>private_key_passphrase</code> in server side</li> <li>specify <code>transport tls</code> and <code>tls_cert_path</code> in cliden side</li> </ul></li> <li>Server certificates signed by self <ul> <li>specify <code>cert_path</code>, <code>private_key_path</code> and <code>private_key_passphrase</code> in server side</li> <li>specify <code>transport tls</code>, <code>tls_cert_path</code> and <code>tls_allow_self_signed_cert true</code> in cliden side</li> </ul></li> <li>Automatically generated server certificates using private CA certs and keys <ul> <li>specify <code>ca_cert_path</code>, <code>ca_private_key_path</code> and <code>ca_private_key_passphrase</code> in server side</li> <li>specify options for cert generation (described below) if needed in server side</li> <li>specify <code>tls_cert_path</code> to specify private CA cert in client side</li> </ul></li> <li>Automatically generated server certificate signed by self (THIS IS ONLY FOR TESTING) <ul> <li>specify <code>insecure true</code>, and options for cert generation (if needed) in server side</li> <li>specify <code>tls_insecure_mode true</code> in client side</li> </ul></li> </ul> <p>Options for cert generation:</p> <ul> <li><code>generate_private_key_length</code> (default: <code>2048</code>)</li> <li><code>generate_cert_country</code> (default: &#39;US&#39;)</li> <li><code>generate_cert_state</code> (default: &#39;CA&#39;)</li> <li><code>generate_cert_locality</code> (default: &#39;Mountain View&#39;)</li> <li><code>generate_cert_common_name</code> (default: hostname of the node)</li> <li><code>generate_cert_expiration</code> (default: <code>3650d</code>)</li> <li><code>generate_cert_digest</code> (default: &#39;sha256&#39;)</li> </ul> <p>This feature is highly experimental. Try this in your development/evaluation environment, and send us feedbacks!</p> <h4>Compatibility between this feature and fluent-plugin-secure-forward</h4> <p>This TLS support feature is almost same with the one of <a href="https://github.com/tagomoris/fluent-plugin-secure-forward">fluent-plugin-secure-forward</a>. Fluentd forward plugins already have authentication feature, <a href="http://www.fluentd.org/blog/fluentd-v0.14.5-has-been-released">introduced at v0.14.5</a>, which is comptible with secure-forward plugins. Now, we have TLS support, so 100% compatible with secure-forward plugin about these protocols.</p> <ul> <li><code>secure_forward</code> output plugin + <code>forward</code> input plugin (w/ TLS, auth): <ul> <li>configure <code>secure_forward</code> output plugin as usual</li> <li>configure <code>forward</code> input plugin using <code>&lt;transport tls&gt;</code> and <code>&lt;secure&gt;</code> sections</li> </ul></li> <li><code>forward</code> output plugin (w/ TLS, auth) + <code>secure_forward</code> input plugin: <ul> <li>configure <code>forward</code> output plugin using <code>transport tls</code> and <code>tls_*</code> options</li> <li>configure <code>forward</code> output plugin with <code>time_as_integer</code> if <code>secure_forward</code> plugin is working on Fluentd v0.12.x or earlier</li> <li>configure <code>secure_forward</code> input plugin as usual</li> </ul></li> </ul> <p>There are some difference about behavior. It will make difference about performance and resource usage:</p> <ul> <li><code>secure_forward</code> output plugin uses connection keep-alive, but <code>forward</code> connects to servers every flush time <ul> <li>establishing TLS connection requires CPU time, so CPU usage might be higher in <code>forward</code> plugins (in both of input and output)</li> <li>keep-alive sometimes make troubles (especially on Internet), so <code>forward</code> may have less network connectivity troubles</li> </ul></li> <li><code>secure_forward</code> output plugin creates threads per connections, but <code>forward</code> uses asynchronous I/O <ul> <li>this may make some difference about CPU time / memory usage, but not sure</li> </ul></li> </ul> <p>Please notice everything to the Fluentd core developer team via issues if you have any troubles.</p> <h3>in_forward: Add source_address_key and source_hostname_key options</h3> <p><code>forward</code> input plugin now has two options to inject fields about data source host address and hostname:</p> <pre><code>&lt;source&gt; @id input_via_network @type forward port 24224 source_address_key address source_hostname_key hostname # All events should have data like: # { &quot;address&quot;: &quot;203.0.113.103&quot;, &quot;hostname&quot;: &quot;web.example.com&quot; } &lt;/source&gt; </code></pre> <p>The Fluentd process with <code>forward</code> input plugin, which is configured to enable these options, will consume much CPU time to manipulate all events. Take care about it.</p> <h3>Fluentd internal log event routing</h3> <p>Fluentd&#39;s internal log events can be caputured with <code>&lt;match fluent.**&gt;</code> sections. Now these events can be routed into the <code>&lt;label @FLUENT_LOG&gt;</code> section if it&#39;s configured.</p> <pre><code>&lt;label @FLUENT_LOG&gt; &lt;match fluent.{warn,error,fatal}&gt; @id fluentd_monitoring @type elasticsearch # .... &lt;/match&gt; # logs in info, debug, trace will be dropped &lt;/label&gt; </code></pre> <p>Make your configuration clean &amp; readable with labels!</p> <h3>Ruby 2.4 support</h3> <p>Fluentd is now tested on Ruby 2.4, and next td-agent will be released with bundled Ruby 2.4.</p> <h3>Major bug fixes</h3> <ul> <li>command line: fix bug to ignore command line options: <code>--rpc-endpoint</code>, <code>--suppress-config-dump</code>, etc <a href="https://github.com/fluent/fluentd/pull/1398">#1398</a></li> <li>supervisor: fix bug of <code>process_name</code> option not to work about supervisor <a href="https://github.com/fluent/fluentd/pull/1380">#1380</a></li> <li>in_forward: Fix a bug not to handle <code>require_ack_response correctly</code> <a href="https://github.com/fluent/fluentd/pull/1389">#1389</a></li> </ul> <p>Enjoy logging!</p> 2017-01-30 <p>Hi users!</p> <p>After the release of Fluentd v0.14.10, we did 2 releases, v0.14.11 at the end of 2016, and v0.14.12 today. Fluentd v0.14.11 release was a kind of quick fix for major bug. And now, we&#39;re very happy to introduce three major new feature with Fluentd v0.14.12!</p> <p>Here are major changes (<a href="https://github.com/fluent/fluentd/blob/master/ChangeLog">full ChangeLog is available here</a>):</p> <p>We say again, fluentd v0.14 is still development version. Newly added features are under experimental... we need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.</p> <h3>Root directory, and no more &quot;path&quot; parameters in buffer configuration</h3> <p>With past versions of Fluentd, file buffer plugin requires <code>path</code> parameter to store buffer chunks in local file systems. These paths should be configured not to use same directories carefully.</p> <pre><code>&lt;match log.service&gt; @id forward_a @type forward &lt;buffer&gt; @type file path /path/of/buffer/for/service flush_interval 1s &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; &lt;match log.system&gt; @id forward_b @type forward &lt;buffer&gt; @type file path /path/of/buffer/for/systems flush_interval 1s &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; </code></pre> <p>With Fluentd v0.14.12, these paths can be configured automatically, using <code>root_dir</code> option in <code>&lt;system&gt;</code> directive.</p> <pre><code>&lt;system&gt; root_dir /path/fluentd/root &lt;/system&gt; &lt;label @traffic&gt; &lt;match log.service&gt; @id forward_a @type forward &lt;buffer&gt; @type file flush_interval 1s &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; &lt;match log.system&gt; @id forward_b @type forward &lt;buffer&gt; @type file flush_interval 1s &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; &lt;/label&gt; </code></pre> <p>There are no need to configure <code>path</code> in <code>&lt;buffer&gt;</code> sections when <code>root_dir</code> is configured and all plugins are configured with <code>@id</code> parameter. All buffer chunks (and storage plugin files) will be stored under <code>/path/fluentd/root/worker0/plugin_id</code>. Don&#39;t miss to monitor the disk usage of that partition!</p> <h3>Multi process workers</h3> <p>Fluentd v0.14.12 supports multi process workers, finally! With this feature, two or more Fluentd workers will be launched, using the same number of CPU cores. This feature can replace <a href="https://github.com/fluent/fluent-plugin-multiprocess">fluent-plugin-multiprocess</a>, with very simple configuration, to process huge traffic on multi CPU servers.</p> <p>Note: There are a requirement: <strong>All configured plugin must support multi process workers feature</strong>.</p> <pre><code>&lt;system&gt; workers 8 # also can be specified by `--workers` command line option root_dir /path/fluentd/root # strongly recommended to specify this &lt;/system&gt; &lt;source&gt; @id input_from_edge @type forward @label @aggr_traffic port 24224 &lt;/source&gt; &lt;label @aggr_traffic&gt; &lt;match **&gt; @id output_to_backend @type forward &lt;buffer&gt; @type file flush_interval 1s chunk_limit_size 8m total_limit_size 64g # limit per plugin instance, per worker (== 64GB * 8 in total max) &lt;/buffer&gt; &lt;server&gt; # ... &lt;/server&gt; &lt;/match&gt; &lt;/label&gt; </code></pre> <p>Eight worker (and a supervisor) processes work like below:</p> <ul> <li>Fluentd launches 8 worker processes in the startup sequence</li> <li>Supervisor process listens <code>port 24224</code></li> <li>When TCP connection requests arrived from client, supervisor delegates <code>accept</code> to a worker of workers (server plugin helper does it)</li> <li>A worker accepts the connection and process inbound traffic from that socket</li> </ul> <p>All worker processes have the set of plugins, routing and buffers (under <code>root_dir/workerN</code> if <code>root_dir</code> is configured), and work in parallel completely. Each workers consumes memory and disk spaces - you should take care to configure buffer spaces, and to monitor memory/disk consumption.</p> <p>This feature is highly experimental. We need many feedbacks, including bug reports, performance graphs, resource consumption summary and more.</p> <h4>Making plugins available with multi process workers</h4> <p>Once again, multi process workers configuration requires all configured plugins support multi process worker feature. It means:</p> <ul> <li>Plugin MUST be implemented with v0.14 API natively <ul> <li>Plugin class is a direct subclass of <code>Fluent::Plugin::Input</code>, <code>Fluent::Plugin::Output</code>, <code>Fluent::Plugin::Filters</code>, etc</li> </ul></li> <li>Plugin MUST have <code>#multi_workers_ready?</code> method and return <code>true</code> <ul> <li>The default return value of this method is <code>true</code> in filter/buffer/parser/formatter/storage plugins</li> <li>But it&#39;s <code>false</code> in input/output plugins</li> <li>3rd party plugins SHOULD re-implemnet <code>#multi_workers_ready?</code> to return <code>true</code> to support multi process workers</li> </ul></li> <li>Plugin MUST NOT use raw sockets to listen network ports <ul> <li>It makes <code>Errno::ADDRINUSE</code> errors with multi process worker configurations</li> <li>Use server plugin helper to create networks servers to listen network port via the supervisor process</li> </ul></li> </ul> <p>Almost all built-in plugins support multi process workers, but <code>in_tail</code> doesn&#39;t support it. If <code>in_tail</code> is configured with <code>workers</code>, Fluentd will fail to start. This disadvantage will be solved in future versions (<a href="https://github.com/fluent/fluentd/issues/1392">see also</a>).</p> <h3>TLS encrypted communication support</h3> <p>There has been many requests to support secure network transportation between Fluentd nodes, for the cases of communication between data centers. Some users had used 3rd party <a href="https://github.com/tagomoris/fluent-plugin-secure-forward">fluent-plugin-secure-forward</a> plugin... But finally, Fluentd core supports it!</p> <p>Here&#39;s a configuration example to use server certificates for server nodes (input side), and to share it in client side (output side) too:</p> <pre><code>##### output side &lt;match secret.log&gt; @id datacenter_a_output @type forward transport tls tls_cert_path /etc/mysecret/cert.pem # server certification file of destination node # or private CA, or ... ### tls_cert_path can be specified with 2 or more paths # tls_cert_path /etc/mysecret/cert1.pem, /etc/mysecret/cert2.pem ### There are some options for detailed configuration.... # tls_version TLSv1_2 # TLS v1.2 (or TLSv1_1) # tls_insecure_mode false # skip all certificate checks if true # tls_allow_self_signed_cert false # tls_verify_hostname true # verify FQDN in certificates, with SNI &lt;server&gt; host dest1.myhost.example.com port 24228 &lt;/server&gt; &lt;server&gt; # or, use host(ip address) &amp; name(hostname to verify with cert) host 203.0.113.103 name dest2.myhost.example.com port 24228 &lt;/server&gt; &lt;/match&gt; ##### input side &lt;source&gt; @id datacenter_b_input @type forward port 24228 &lt;transport tls&gt; # version TLSv1_2 cert_path /etc/mysecret/cert1 private_key_path /etc/mysecret/server.key.pem private_key_passphrase my_secret_passphrase_for_key &lt;/transport&gt; &lt;/source&gt; </code></pre> <p>If your server certificate requires intermediate CA certificates to be verified, concat these certs to make chained certs (just like nginx) and specify it in <code>cert_path</code>.</p> <p>There are some patterns to configure <code>forward</code> input plugin with TLS. All parameters for input plugin (server side) below should be specified in <code>&lt;transport tls&gt;</code> section.</p> <ul> <li>Server certificates signed by public CA <ul> <li>specify <code>cert_path</code>, <code>private_key_path</code> and <code>private_key_passphrase</code> in server side</li> <li>just specify <code>transport tls</code> in clien side (if your system&#39;s cert store has valid root CA certs)</li> </ul></li> <li>Server certificates signed by private CA <ul> <li>specify <code>cert_path</code>, <code>private_key_path</code> and <code>private_key_passphrase</code> in server side</li> <li>specify <code>transport tls</code> and <code>tls_cert_path</code> in cliden side</li> </ul></li> <li>Server certificates signed by self <ul> <li>specify <code>cert_path</code>, <code>private_key_path</code> and <code>private_key_passphrase</code> in server side</li> <li>specify <code>transport tls</code>, <code>tls_cert_path</code> and <code>tls_allow_self_signed_cert true</code> in cliden side</li> </ul></li> <li>Automatically generated server certificates using private CA certs and keys <ul> <li>specify <code>ca_cert_path</code>, <code>ca_private_key_path</code> and <code>ca_private_key_passphrase</code> in server side</li> <li>specify options for cert generation (described below) if needed in server side</li> <li>specify <code>tls_cert_path</code> to specify private CA cert in client side</li> </ul></li> <li>Automatically generated server certificate signed by self (THIS IS ONLY FOR TESTING) <ul> <li>specify <code>insecure true</code>, and options for cert generation (if needed) in server side</li> <li>specify <code>tls_insecure_mode true</code> in client side</li> </ul></li> </ul> <p>Options for cert generation:</p> <ul> <li><code>generate_private_key_length</code> (default: <code>2048</code>)</li> <li><code>generate_cert_country</code> (default: &#39;US&#39;)</li> <li><code>generate_cert_state</code> (default: &#39;CA&#39;)</li> <li><code>generate_cert_locality</code> (default: &#39;Mountain View&#39;)</li> <li><code>generate_cert_common_name</code> (default: hostname of the node)</li> <li><code>generate_cert_expiration</code> (default: <code>3650d</code>)</li> <li><code>generate_cert_digest</code> (default: &#39;sha256&#39;)</li> </ul> <p>This feature is highly experimental. Try this in your development/evaluation environment, and send us feedbacks!</p> <h4>Compatibility between this feature and fluent-plugin-secure-forward</h4> <p>This TLS support feature is almost same with the one of <a href="https://github.com/tagomoris/fluent-plugin-secure-forward">fluent-plugin-secure-forward</a>. Fluentd forward plugins already have authentication feature, <a href="http://www.fluentd.org/blog/fluentd-v0.14.5-has-been-released">introduced at v0.14.5</a>, which is comptible with secure-forward plugins. Now, we have TLS support, so 100% compatible with secure-forward plugin about these protocols.</p> <ul> <li><code>secure_forward</code> output plugin + <code>forward</code> input plugin (w/ TLS, auth): <ul> <li>configure <code>secure_forward</code> output plugin as usual</li> <li>configure <code>forward</code> input plugin using <code>&lt;transport tls&gt;</code> and <code>&lt;secure&gt;</code> sections</li> </ul></li> <li><code>forward</code> output plugin (w/ TLS, auth) + <code>secure_forward</code> input plugin: <ul> <li>configure <code>forward</code> output plugin using <code>transport tls</code> and <code>tls_*</code> options</li> <li>configure <code>forward</code> output plugin with <code>time_as_integer</code> if <code>secure_forward</code> plugin is working on Fluentd v0.12.x or earlier</li> <li>configure <code>secure_forward</code> input plugin as usual</li> </ul></li> </ul> <p>There are some difference about behavior. It will make difference about performance and resource usage:</p> <ul> <li><code>secure_forward</code> output plugin uses connection keep-alive, but <code>forward</code> connects to servers every flush time <ul> <li>establishing TLS connection requires CPU time, so CPU usage might be higher in <code>forward</code> plugins (in both of input and output)</li> <li>keep-alive sometimes make troubles (especially on Internet), so <code>forward</code> may have less network connectivity troubles</li> </ul></li> <li><code>secure_forward</code> output plugin creates threads per connections, but <code>forward</code> uses asynchronous I/O <ul> <li>this may make some difference about CPU time / memory usage, but not sure</li> </ul></li> </ul> <p>Please notice everything to the Fluentd core developer team via issues if you have any troubles.</p> <h3>in_forward: Add source_address_key and source_hostname_key options</h3> <p><code>forward</code> input plugin now has two options to inject fields about data source host address and hostname:</p> <pre><code>&lt;source&gt; @id input_via_network @type forward port 24224 source_address_key address source_hostname_key hostname # All events should have data like: # { &quot;address&quot;: &quot;203.0.113.103&quot;, &quot;hostname&quot;: &quot;web.example.com&quot; } &lt;/source&gt; </code></pre> <p>The Fluentd process with <code>forward</code> input plugin, which is configured to enable these options, will consume much CPU time to manipulate all events. Take care about it.</p> <h3>Fluentd internal log event routing</h3> <p>Fluentd&#39;s internal log events can be caputured with <code>&lt;match fluent.**&gt;</code> sections. Now these events can be routed into the <code>&lt;label @FLUENT_LOG&gt;</code> section if it&#39;s configured.</p> <pre><code>&lt;label @FLUENT_LOG&gt; &lt;match fluent.{warn,error,fatal}&gt; @id fluentd_monitoring @type elasticsearch # .... &lt;/match&gt; # logs in info, debug, trace will be dropped &lt;/label&gt; </code></pre> <p>Make your configuration clean &amp; readable with labels!</p> <h3>Ruby 2.4 support</h3> <p>Fluentd is now tested on Ruby 2.4, and next td-agent will be released with bundled Ruby 2.4.</p> <h3>Major bug fixes</h3> <ul> <li>command line: fix bug to ignore command line options: <code>--rpc-endpoint</code>, <code>--suppress-config-dump</code>, etc <a href="https://github.com/fluent/fluentd/pull/1398">#1398</a></li> <li>supervisor: fix bug of <code>process_name</code> option not to work about supervisor <a href="https://github.com/fluent/fluentd/pull/1380">#1380</a></li> <li>in_forward: Fix a bug not to handle <code>require_ack_response correctly</code> <a href="https://github.com/fluent/fluentd/pull/1389">#1389</a></li> </ul> <p>Enjoy logging!</p>