Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v1.7.0 has been released https://www.fluentd.org/blog/fluentd-v1.7.0-has-been-released <p>Hi users!</p> <p>We have released v1.7.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new buffer plugin and some improvement.</p> <h3>Add <code>file_single</code> buffer plugin</h3> <p>This new buffer plugin aims to reduce the number of IO operation for high traffic environment. <code>buf_file_single</code> doesn&#39;t have <code>.meta</code> file and it embeds the chunk key value to the file path like v0.12&#39;s <code>buf_file</code>.</p> <pre><code>&lt;match pattern&gt; @type forward &lt;buffer tag&gt; @type file_single path /path/to/buffer/test &lt;/buffer&gt; # other parameters... &lt;/match&gt; </code></pre> <p>With this configuration, here is buffer path example with <code>test.log</code> event tag.</p> <pre><code># format is /$path_parameter/fsb.$tag.$chunk_id.buf /path/to/buffer/test/fsb.test.log.b513b61c9791029c2513b61c9791029c2.buf </code></pre> <p><code>buf_file_single</code> now has one limitation. chunk keys must be tag or one field key.</p> <pre><code>&lt;buffer tag&gt; # OK &lt;buffer key&gt; # OK &lt;buffer time&gt; # NG &lt;buffer key1,key2&gt; # NG </code></pre> <p>We will remove this limitation by adding metadata header in the file.</p> <h3>Add <code>http</code> output plugin</h3> <p>You can now send data to your destination via HTTP/HTTPS.</p> <p>See <a href="https://docs.fluentd.org/output/http">out_http article</a> in the document.</p> <h3>formatter_csv/parser_csv: Improve the performance</h3> <p><code>formatter_csv</code> is now 2x faster by avoiding ruby&#39;s CSV object per event.</p> <p><code>parser_csv</code> introduces <code>parser_type</code> parameter to change the internal parser. If set <code>parser_type fast</code>, <code>parser_csv</code> uses own fast parser. The drawback of fast parser is not robust unlike ruby&#39;s CSV parser. So if you use fast parser with broken CSV, it may generate broken record. You need to check your CSV is correct before use fast parser.</p> <h3>in_tcp: Emit multiple events at a time</h3> <p>Previous <code>in_tcp</code> emits events one by one even when accepts multiple events in received data. <code>in_tcp</code> now emits multiple events at a time. This reduces CPU usage because <code>emit</code> is heavy task.</p> <h3>core: Improve several points</h3> <ul> <li><code>Fluent::EventTime.now</code> uses 2x faster implementation</li> <li>Remove unncessary data from buffer/output </li> <li>Avoid unnessary object allocation</li> </ul> <p>This may reduce CPU usage, a few percent.</p> <h3>Major bug fixes</h3> <ul> <li>output: Fix data lost on decompression by EOF check bug</li> <li>in_tail: Don&#39;t call parser&#39;s configure twice. This fixes the regression of configuration error.</li> </ul> <p>Enjoy logging!</p> 2019-08-21 <p>Hi users!</p> <p>We have released v1.7.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new buffer plugin and some improvement.</p> <h3>Add <code>file_single</code> buffer plugin</h3> <p>This new buffer plugin aims to reduce the number of IO operation for high traffic environment. <code>buf_file_single</code> doesn&#39;t have <code>.meta</code> file and it embeds the chunk key value to the file path like v0.12&#39;s <code>buf_file</code>.</p> <pre><code>&lt;match pattern&gt; @type forward &lt;buffer tag&gt; @type file_single path /path/to/buffer/test &lt;/buffer&gt; # other parameters... &lt;/match&gt; </code></pre> <p>With this configuration, here is buffer path example with <code>test.log</code> event tag.</p> <pre><code># format is /$path_parameter/fsb.$tag.$chunk_id.buf /path/to/buffer/test/fsb.test.log.b513b61c9791029c2513b61c9791029c2.buf </code></pre> <p><code>buf_file_single</code> now has one limitation. chunk keys must be tag or one field key.</p> <pre><code>&lt;buffer tag&gt; # OK &lt;buffer key&gt; # OK &lt;buffer time&gt; # NG &lt;buffer key1,key2&gt; # NG </code></pre> <p>We will remove this limitation by adding metadata header in the file.</p> <h3>Add <code>http</code> output plugin</h3> <p>You can now send data to your destination via HTTP/HTTPS.</p> <p>See <a href="https://docs.fluentd.org/output/http">out_http article</a> in the document.</p> <h3>formatter_csv/parser_csv: Improve the performance</h3> <p><code>formatter_csv</code> is now 2x faster by avoiding ruby&#39;s CSV object per event.</p> <p><code>parser_csv</code> introduces <code>parser_type</code> parameter to change the internal parser. If set <code>parser_type fast</code>, <code>parser_csv</code> uses own fast parser. The drawback of fast parser is not robust unlike ruby&#39;s CSV parser. So if you use fast parser with broken CSV, it may generate broken record. You need to check your CSV is correct before use fast parser.</p> <h3>in_tcp: Emit multiple events at a time</h3> <p>Previous <code>in_tcp</code> emits events one by one even when accepts multiple events in received data. <code>in_tcp</code> now emits multiple events at a time. This reduces CPU usage because <code>emit</code> is heavy task.</p> <h3>core: Improve several points</h3> <ul> <li><code>Fluent::EventTime.now</code> uses 2x faster implementation</li> <li>Remove unncessary data from buffer/output </li> <li>Avoid unnessary object allocation</li> </ul> <p>This may reduce CPU usage, a few percent.</p> <h3>Major bug fixes</h3> <ul> <li>output: Fix data lost on decompression by EOF check bug</li> <li>in_tail: Don&#39;t call parser&#39;s configure twice. This fixes the regression of configuration error.</li> </ul> <p>Enjoy logging!</p> Fluentd v1.6.3 has been released https://www.fluentd.org/blog/fluentd-v1.6.3-has-been-released <p>Hi users!</p> <p>We have released v1.6.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. These release includes several enhancements.</p> <h3>in_syslog: Add <code>emit_unmatched_lines</code> parameter</h3> <p>This parameter enables you to get unparseable log as an event. This is same as in_tail&#39;s one.</p> <p>If <code>true</code>, emitted event is like below:</p> <pre><code>tag: syslog.unmatched # &#39;syslog&#39; depends on tag parameter time: parsed time record: {&quot;unmatched_line&quot;:&quot;broken line&quot;} </code></pre> <h3>buf_file: Add <code>path_suffix</code> parameter</h3> <p>You can change the suffix of buffer file. Here is an example:</p> <pre><code># default /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta # with &#39;path_suffix .buf&#39; /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf.meta </code></pre> <p>This parameter is useful when <code>.log</code> suffix is not fit for your system, </p> <h3>Bug fixes</h3> <ul> <li>http_server helper: Fix method re-define warning</li> </ul> <p>Enjoy logging!</p> 2019-07-29 <p>Hi users!</p> <p>We have released v1.6.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. These release includes several enhancements.</p> <h3>in_syslog: Add <code>emit_unmatched_lines</code> parameter</h3> <p>This parameter enables you to get unparseable log as an event. This is same as in_tail&#39;s one.</p> <p>If <code>true</code>, emitted event is like below:</p> <pre><code>tag: syslog.unmatched # &#39;syslog&#39; depends on tag parameter time: parsed time record: {&quot;unmatched_line&quot;:&quot;broken line&quot;} </code></pre> <h3>buf_file: Add <code>path_suffix</code> parameter</h3> <p>You can change the suffix of buffer file. Here is an example:</p> <pre><code># default /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta # with &#39;path_suffix .buf&#39; /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf.meta </code></pre> <p>This parameter is useful when <code>.log</code> suffix is not fit for your system, </p> <h3>Bug fixes</h3> <ul> <li>http_server helper: Fix method re-define warning</li> </ul> <p>Enjoy logging!</p> Fluentd v1.6.1 and v1.6.2 has been released https://www.fluentd.org/blog/fluentd-v1.6.2-has-been-released <p>Hi users!</p> <p>We have released v1.6.1 and v1.6.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. These release includes several enhancements and bug fixes.</p> <h3>http_server helper: Add title argument to the API</h3> <p>This fix is important to launch HTTP servers in different plugins. Old implementation uses same thread identifier for all HTTP servers and it won&#39;t work.</p> <p>Check also <a href="https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-http_server#create_http_server-title-addr-port-logger-default_app-nil-and-block">API article</a>.</p> <h3>certificates: Support all private keys OpenSSL supports</h3> <p>fluentd assumes private key is RSA before. Now, other types are supported, e.g. EC and DSA.</p> <h3>Other bug fixes</h3> <ul> <li>plugin<em>config</em>formatter: use new doc URL</li> <li>out_forward: Avoid zero division error when there are no available nodese</li> </ul> <p>Enjoy logging!</p> 2019-07-11 <p>Hi users!</p> <p>We have released v1.6.1 and v1.6.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. These release includes several enhancements and bug fixes.</p> <h3>http_server helper: Add title argument to the API</h3> <p>This fix is important to launch HTTP servers in different plugins. Old implementation uses same thread identifier for all HTTP servers and it won&#39;t work.</p> <p>Check also <a href="https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-http_server#create_http_server-title-addr-port-logger-default_app-nil-and-block">API article</a>.</p> <h3>certificates: Support all private keys OpenSSL supports</h3> <p>fluentd assumes private key is RSA before. Now, other types are supported, e.g. EC and DSA.</p> <h3>Other bug fixes</h3> <ul> <li>plugin<em>config</em>formatter: use new doc URL</li> <li>out_forward: Avoid zero division error when there are no available nodese</li> </ul> <p>Enjoy logging!</p> Fluentd v1.6.0 has been released https://www.fluentd.org/blog/fluentd-v1.6.0-has-been-released <p>Hi users!</p> <p>We have released v1.6.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new plugin helper and some improvement.</p> <h3>http_server plugin helper</h3> <p>This plugin helper is for <code>in_monitor_agent</code> like plugins. Here is code example:</p> <pre><code>module Fluent::Plugin class FooServerInput &lt; Input Fluent::Plugin.register_input(&#39;foo_server&#39;, self) helpers :http_server # ... def start super create_http_server(addr: @bind, port: @port, logger: log) do |serv| serv.get(&#39;/api/hello&#39;) { |req| # Return value is an Array which consists of status code, headers and body [200, {&#39;Content-Type&#39; =&gt; &#39;text/plain&#39;}, &#39;hello get&#39;] } serv.post(&#39;/api/hello&#39;) { |req| [200, {&#39;Content-Type&#39; =&gt; &#39;text/plain&#39;}, &#39;hello post&#39;] } # support other HTTP method end end end end </code></pre> <p>This helper uses WEBrick by default. But if you install <code>async-http</code> gem, use <code>async-http</code> instead for the performance. <code>in_monitor_agent</code> replaced HTTP server implemenation with <code>http_server</code> helper.</p> <h3>in_monitor_agent: Add more buffer/output metrics</h3> <p>Since v1.6.0, <code>monitor_agent</code> input plugin exposes more buffer/output metrics. Here is the comparison.</p> <pre><code># v1.6 { &quot;plugin_id&quot;: &quot;out_forward&quot;, &quot;plugin_category&quot;: &quot;output&quot;, &quot;type&quot;: &quot;forward&quot;, &quot;output_plugin&quot;: true, &quot;buffer_queue_length&quot;: 0, &quot;buffer_timekeys&quot;: [], &quot;buffer_total_queued_size&quot;: 0, &quot;retry_count&quot;: 0, &quot;emit_records&quot;: 0, &quot;emit_count&quot;: 0, &quot;write_count&quot;: 0, &quot;rollback_count&quot;: 0, &quot;slow_flush_count&quot;: 0, &quot;flush_time_count&quot;: 0, &quot;buffer_stage_length&quot;: 0, &quot;buffer_stage_byte_size&quot;: 0, &quot;buffer_queue_byte_size&quot;: 0, &quot;buffer_available_buffer_space_ratios&quot;: 100, &quot;retry&quot;: {} } # v1.5 or earlier { &quot;plugin_id&quot;: &quot;out_forward&quot;, &quot;plugin_category&quot;: &quot;output&quot;, &quot;type&quot;: &quot;forward&quot;, &quot;output_plugin&quot;: true, &quot;buffer_queue_length&quot;: 0, &quot;buffer_timekeys&quot;: [], &quot;buffer_total_queued_size&quot;: 0, &quot;retry_count&quot;: 0, &quot;retry&quot;: {} } </code></pre> <p>New fields are below:</p> <ul> <li><code>emit_records</code>: The total number of emitted records</li> <li><code>emit_count</code>: The total number of <code>emit</code> call in output plugin</li> <li><code>write_count</code>: The total number of <code>write/try_write</code> call in output plugin</li> <li><code>rollback_count</code>: The total number of rollback. rollback happens when <code>write/try_write</code> failed</li> <li><code>slow_flush_count</code>: The total number of slow flush. This count will be incremented when buffer flush is longer than <code>slow_flush_log_threshold</code></li> <li><code>flush_time_count</code>: The total time of buffer flush in milliseconds</li> <li><code>buffer_stage_length</code>: Current lenght of staged buffer chunks</li> <li><code>buffer_stage_byte_size</code>: Current bytesize of staged buffer chunks</li> <li><code>buffer_queue_byte_size</code>: Current bytesize of queued buffer chunks</li> <li><code>buffer_available_buffer_space_ratios</code>: Show available space for buffer</li> </ul> <h3>socket/out_forward: Add <code>connect_timeout</code> parameter for TCP/TLS</h3> <p>This socket timeout is used for connection establishment. If your DNS returns wrong IP or machine, connection establishment waits a long time, e.g. 180 seconds by system default. <code>connect_timeout</code> parameter avoids this kind of problem.</p> <h3>Major bug fixes</h3> <ul> <li>cert: Fix cert pattern to accept certificates which don&#39;t end with newline</li> </ul> <p>Enjoy logging!</p> 2019-07-02 <p>Hi users!</p> <p>We have released v1.6.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new plugin helper and some improvement.</p> <h3>http_server plugin helper</h3> <p>This plugin helper is for <code>in_monitor_agent</code> like plugins. Here is code example:</p> <pre><code>module Fluent::Plugin class FooServerInput &lt; Input Fluent::Plugin.register_input(&#39;foo_server&#39;, self) helpers :http_server # ... def start super create_http_server(addr: @bind, port: @port, logger: log) do |serv| serv.get(&#39;/api/hello&#39;) { |req| # Return value is an Array which consists of status code, headers and body [200, {&#39;Content-Type&#39; =&gt; &#39;text/plain&#39;}, &#39;hello get&#39;] } serv.post(&#39;/api/hello&#39;) { |req| [200, {&#39;Content-Type&#39; =&gt; &#39;text/plain&#39;}, &#39;hello post&#39;] } # support other HTTP method end end end end </code></pre> <p>This helper uses WEBrick by default. But if you install <code>async-http</code> gem, use <code>async-http</code> instead for the performance. <code>in_monitor_agent</code> replaced HTTP server implemenation with <code>http_server</code> helper.</p> <h3>in_monitor_agent: Add more buffer/output metrics</h3> <p>Since v1.6.0, <code>monitor_agent</code> input plugin exposes more buffer/output metrics. Here is the comparison.</p> <pre><code># v1.6 { &quot;plugin_id&quot;: &quot;out_forward&quot;, &quot;plugin_category&quot;: &quot;output&quot;, &quot;type&quot;: &quot;forward&quot;, &quot;output_plugin&quot;: true, &quot;buffer_queue_length&quot;: 0, &quot;buffer_timekeys&quot;: [], &quot;buffer_total_queued_size&quot;: 0, &quot;retry_count&quot;: 0, &quot;emit_records&quot;: 0, &quot;emit_count&quot;: 0, &quot;write_count&quot;: 0, &quot;rollback_count&quot;: 0, &quot;slow_flush_count&quot;: 0, &quot;flush_time_count&quot;: 0, &quot;buffer_stage_length&quot;: 0, &quot;buffer_stage_byte_size&quot;: 0, &quot;buffer_queue_byte_size&quot;: 0, &quot;buffer_available_buffer_space_ratios&quot;: 100, &quot;retry&quot;: {} } # v1.5 or earlier { &quot;plugin_id&quot;: &quot;out_forward&quot;, &quot;plugin_category&quot;: &quot;output&quot;, &quot;type&quot;: &quot;forward&quot;, &quot;output_plugin&quot;: true, &quot;buffer_queue_length&quot;: 0, &quot;buffer_timekeys&quot;: [], &quot;buffer_total_queued_size&quot;: 0, &quot;retry_count&quot;: 0, &quot;retry&quot;: {} } </code></pre> <p>New fields are below:</p> <ul> <li><code>emit_records</code>: The total number of emitted records</li> <li><code>emit_count</code>: The total number of <code>emit</code> call in output plugin</li> <li><code>write_count</code>: The total number of <code>write/try_write</code> call in output plugin</li> <li><code>rollback_count</code>: The total number of rollback. rollback happens when <code>write/try_write</code> failed</li> <li><code>slow_flush_count</code>: The total number of slow flush. This count will be incremented when buffer flush is longer than <code>slow_flush_log_threshold</code></li> <li><code>flush_time_count</code>: The total time of buffer flush in milliseconds</li> <li><code>buffer_stage_length</code>: Current lenght of staged buffer chunks</li> <li><code>buffer_stage_byte_size</code>: Current bytesize of staged buffer chunks</li> <li><code>buffer_queue_byte_size</code>: Current bytesize of queued buffer chunks</li> <li><code>buffer_available_buffer_space_ratios</code>: Show available space for buffer</li> </ul> <h3>socket/out_forward: Add <code>connect_timeout</code> parameter for TCP/TLS</h3> <p>This socket timeout is used for connection establishment. If your DNS returns wrong IP or machine, connection establishment waits a long time, e.g. 180 seconds by system default. <code>connect_timeout</code> parameter avoids this kind of problem.</p> <h3>Major bug fixes</h3> <ul> <li>cert: Fix cert pattern to accept certificates which don&#39;t end with newline</li> </ul> <p>Enjoy logging!</p> Fluentd v1.5.2 has been released https://www.fluentd.org/blog/fluentd-v1.5.2-has-been-released <p>Hi users!</p> <p>We have released v1.5.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is for fixing <code>out_forward</code> keepalive issue. If you want to use <code>keepalive</code> and <code>&lt;security&gt;</code>, update fluentd to v1.5.2.</p> <p>Enjoy logging!</p> 2019-06-14 <p>Hi users!</p> <p>We have released v1.5.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is for fixing <code>out_forward</code> keepalive issue. If you want to use <code>keepalive</code> and <code>&lt;security&gt;</code>, update fluentd to v1.5.2.</p> <p>Enjoy logging!</p>