Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v1.7.3 has been released https://www.fluentd.org/blog/fluentd-v1.7.3-has-been-released <p>Hi users!</p> <p>We have released v1.7.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is mainly for fixing socket cache bug of out_forward.</p> <h3>in_syslog: Replace <code>priority_key</code> with <code>severity_key</code></h3> <p><code>in_syslog</code>&#39;s <code>priority_key</code> parameter is misleading name because it sets severity, not priority value. So we add <code>severity_key</code> parameter.</p> <pre><code>&lt;source&gt; @type syslog severity_key severity tag syslog &lt;/source&gt; </code></pre> <p><code>priority_key</code> is still supported for existing users but we will remove <code>priority_key</code> parameter at fluentd v2.</p> <h3>Major bug fixes</h3> <ul> <li>out_forward: Fix <code>nil</code> error after purge obsoleted socket in socket cache</li> </ul> <p>Enjoy logging!</p> 2019-10-01 <p>Hi users!</p> <p>We have released v1.7.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is mainly for fixing socket cache bug of out_forward.</p> <h3>in_syslog: Replace <code>priority_key</code> with <code>severity_key</code></h3> <p><code>in_syslog</code>&#39;s <code>priority_key</code> parameter is misleading name because it sets severity, not priority value. So we add <code>severity_key</code> parameter.</p> <pre><code>&lt;source&gt; @type syslog severity_key severity tag syslog &lt;/source&gt; </code></pre> <p><code>priority_key</code> is still supported for existing users but we will remove <code>priority_key</code> parameter at fluentd v2.</p> <h3>Major bug fixes</h3> <ul> <li>out_forward: Fix <code>nil</code> error after purge obsoleted socket in socket cache</li> </ul> <p>Enjoy logging!</p> Fluentd v1.7.2 has been released https://www.fluentd.org/blog/fluentd-v1.7.2-has-been-released <p>Hi users!</p> <p>We have released v1.7.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several fixes.</p> <h3>in_tcp: Add <code>security/client</code> to restrict access</h3> <p>This is similar to <code>in_forward</code>. You can limit the client by host/network. Here is configuration example:</p> <pre><code>&lt;source&gt; @type tcp # other parameters... &lt;security&gt; &lt;client&gt; host app_server1.com &lt;/client&gt; &lt;client&gt; host web_server1.com &lt;/client&gt; &lt;/security&gt; &lt;/source&gt; </code></pre> <p>With this configuration, <code>in_tcp</code> rejects the data from other hosts, e.g. <code>app_server2.com</code>.</p> <h3>Major bug fixes</h3> <ul> <li>buf_file/buf_file_single: fix to handle compress data during restart</li> <li>plugin: Use <code>__send__</code> to avoid conflict with user defined <code>send</code> method</li> </ul> <p>Enjoy logging!</p> 2019-09-19 <p>Hi users!</p> <p>We have released v1.7.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several fixes.</p> <h3>in_tcp: Add <code>security/client</code> to restrict access</h3> <p>This is similar to <code>in_forward</code>. You can limit the client by host/network. Here is configuration example:</p> <pre><code>&lt;source&gt; @type tcp # other parameters... &lt;security&gt; &lt;client&gt; host app_server1.com &lt;/client&gt; &lt;client&gt; host web_server1.com &lt;/client&gt; &lt;/security&gt; &lt;/source&gt; </code></pre> <p>With this configuration, <code>in_tcp</code> rejects the data from other hosts, e.g. <code>app_server2.com</code>.</p> <h3>Major bug fixes</h3> <ul> <li>buf_file/buf_file_single: fix to handle compress data during restart</li> <li>plugin: Use <code>__send__</code> to avoid conflict with user defined <code>send</code> method</li> </ul> <p>Enjoy logging!</p> Fluentd v1.7.1 has been released https://www.fluentd.org/blog/fluentd-v1.7.1-has-been-released <p>Hi users!</p> <p>We have released v1.7.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements and fixes.</p> <h3>parser_syslog: Add <code>string</code> parser for rfc3164 message</h3> <p>v1.7.1 introduces <code>parser_type</code> parameter to choose internal implementation for <code>rfc3164</code> message.</p> <p><code>parser_syslog</code> uses <code>regexp</code> parser to parse rfc3164 message by default but it doesn&#39;t work for some patterns, see <a href="https://github.com/fluent/fluentd/issues/2585">this issue</a>. <code>string</code> parser avoids this problem.</p> <p>Here is a configuration example:</p> <pre><code>&lt;source&gt; @type syslog &lt;parse&gt; parser_type string &lt;/buffer&gt; # other parameters... &lt;/source&gt; </code></pre> <p><code>string</code> parser is 2x faster than <code>regexp</code> parser, so we recommend to use <code>string</code> parser for new deployment.</p> <h3>Major bug fixes</h3> <ul> <li>buf_file/buf_file_single: fix to ignore placeholder based path during resume</li> <li>buf_file: Ensure to remove metadata file after buffer creation failure</li> <li>server helper: Ignore <code>ETIMEDOUT</code> error in <code>SSL_accept</code></li> </ul> <p>Enjoy logging!</p> 2019-09-08 <p>Hi users!</p> <p>We have released v1.7.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements and fixes.</p> <h3>parser_syslog: Add <code>string</code> parser for rfc3164 message</h3> <p>v1.7.1 introduces <code>parser_type</code> parameter to choose internal implementation for <code>rfc3164</code> message.</p> <p><code>parser_syslog</code> uses <code>regexp</code> parser to parse rfc3164 message by default but it doesn&#39;t work for some patterns, see <a href="https://github.com/fluent/fluentd/issues/2585">this issue</a>. <code>string</code> parser avoids this problem.</p> <p>Here is a configuration example:</p> <pre><code>&lt;source&gt; @type syslog &lt;parse&gt; parser_type string &lt;/buffer&gt; # other parameters... &lt;/source&gt; </code></pre> <p><code>string</code> parser is 2x faster than <code>regexp</code> parser, so we recommend to use <code>string</code> parser for new deployment.</p> <h3>Major bug fixes</h3> <ul> <li>buf_file/buf_file_single: fix to ignore placeholder based path during resume</li> <li>buf_file: Ensure to remove metadata file after buffer creation failure</li> <li>server helper: Ignore <code>ETIMEDOUT</code> error in <code>SSL_accept</code></li> </ul> <p>Enjoy logging!</p> Fluentd v1.7.0 has been released https://www.fluentd.org/blog/fluentd-v1.7.0-has-been-released <p>Hi users!</p> <p>We have released v1.7.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new buffer plugin and some improvement.</p> <h3>Add <code>file_single</code> buffer plugin</h3> <p>This new buffer plugin aims to reduce the number of IO operation for high traffic environment. <code>buf_file_single</code> doesn&#39;t have <code>.meta</code> file and it embeds the chunk key value to the file path like v0.12&#39;s <code>buf_file</code>.</p> <pre><code>&lt;match pattern&gt; @type forward &lt;buffer tag&gt; @type file_single path /path/to/buffer/test &lt;/buffer&gt; # other parameters... &lt;/match&gt; </code></pre> <p>With this configuration, here is buffer path example with <code>test.log</code> event tag.</p> <pre><code># format is /$path_parameter/fsb.$tag.$chunk_id.buf /path/to/buffer/test/fsb.test.log.b513b61c9791029c2513b61c9791029c2.buf </code></pre> <p><code>buf_file_single</code> now has one limitation. chunk keys must be tag or one field key.</p> <pre><code>&lt;buffer tag&gt; # OK &lt;buffer key&gt; # OK &lt;buffer time&gt; # NG &lt;buffer key1,key2&gt; # NG </code></pre> <p>We will remove this limitation by adding metadata header in the file.</p> <h3>Add <code>http</code> output plugin</h3> <p>You can now send data to your destination via HTTP/HTTPS.</p> <p>See <a href="https://docs.fluentd.org/output/http">out_http article</a> in the document.</p> <h3>formatter_csv/parser_csv: Improve the performance</h3> <p><code>formatter_csv</code> is now 2x faster by avoiding ruby&#39;s CSV object per event.</p> <p><code>parser_csv</code> introduces <code>parser_type</code> parameter to change the internal parser. If set <code>parser_type fast</code>, <code>parser_csv</code> uses own fast parser. The drawback of fast parser is not robust unlike ruby&#39;s CSV parser. So if you use fast parser with broken CSV, it may generate broken record. You need to check your CSV is correct before use fast parser.</p> <h3>in_tcp: Emit multiple events at a time</h3> <p>Previous <code>in_tcp</code> emits events one by one even when accepts multiple events in received data. <code>in_tcp</code> now emits multiple events at a time. This reduces CPU usage because <code>emit</code> is heavy task.</p> <h3>core: Improve several points</h3> <ul> <li><code>Fluent::EventTime.now</code> uses 2x faster implementation</li> <li>Remove unncessary data from buffer/output </li> <li>Avoid unnessary object allocation</li> </ul> <p>This may reduce CPU usage, a few percent.</p> <h3>Major bug fixes</h3> <ul> <li>output: Fix data lost on decompression by EOF check bug</li> <li>in_tail: Don&#39;t call parser&#39;s configure twice. This fixes the regression of configuration error.</li> </ul> <p>Enjoy logging!</p> 2019-08-21 <p>Hi users!</p> <p>We have released v1.7.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new buffer plugin and some improvement.</p> <h3>Add <code>file_single</code> buffer plugin</h3> <p>This new buffer plugin aims to reduce the number of IO operation for high traffic environment. <code>buf_file_single</code> doesn&#39;t have <code>.meta</code> file and it embeds the chunk key value to the file path like v0.12&#39;s <code>buf_file</code>.</p> <pre><code>&lt;match pattern&gt; @type forward &lt;buffer tag&gt; @type file_single path /path/to/buffer/test &lt;/buffer&gt; # other parameters... &lt;/match&gt; </code></pre> <p>With this configuration, here is buffer path example with <code>test.log</code> event tag.</p> <pre><code># format is /$path_parameter/fsb.$tag.$chunk_id.buf /path/to/buffer/test/fsb.test.log.b513b61c9791029c2513b61c9791029c2.buf </code></pre> <p><code>buf_file_single</code> now has one limitation. chunk keys must be tag or one field key.</p> <pre><code>&lt;buffer tag&gt; # OK &lt;buffer key&gt; # OK &lt;buffer time&gt; # NG &lt;buffer key1,key2&gt; # NG </code></pre> <p>We will remove this limitation by adding metadata header in the file.</p> <h3>Add <code>http</code> output plugin</h3> <p>You can now send data to your destination via HTTP/HTTPS.</p> <p>See <a href="https://docs.fluentd.org/output/http">out_http article</a> in the document.</p> <h3>formatter_csv/parser_csv: Improve the performance</h3> <p><code>formatter_csv</code> is now 2x faster by avoiding ruby&#39;s CSV object per event.</p> <p><code>parser_csv</code> introduces <code>parser_type</code> parameter to change the internal parser. If set <code>parser_type fast</code>, <code>parser_csv</code> uses own fast parser. The drawback of fast parser is not robust unlike ruby&#39;s CSV parser. So if you use fast parser with broken CSV, it may generate broken record. You need to check your CSV is correct before use fast parser.</p> <h3>in_tcp: Emit multiple events at a time</h3> <p>Previous <code>in_tcp</code> emits events one by one even when accepts multiple events in received data. <code>in_tcp</code> now emits multiple events at a time. This reduces CPU usage because <code>emit</code> is heavy task.</p> <h3>core: Improve several points</h3> <ul> <li><code>Fluent::EventTime.now</code> uses 2x faster implementation</li> <li>Remove unncessary data from buffer/output </li> <li>Avoid unnessary object allocation</li> </ul> <p>This may reduce CPU usage, a few percent.</p> <h3>Major bug fixes</h3> <ul> <li>output: Fix data lost on decompression by EOF check bug</li> <li>in_tail: Don&#39;t call parser&#39;s configure twice. This fixes the regression of configuration error.</li> </ul> <p>Enjoy logging!</p> Fluentd v1.6.3 has been released https://www.fluentd.org/blog/fluentd-v1.6.3-has-been-released <p>Hi users!</p> <p>We have released v1.6.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. These release includes several enhancements.</p> <h3>in_syslog: Add <code>emit_unmatched_lines</code> parameter</h3> <p>This parameter enables you to get unparseable log as an event. This is same as in_tail&#39;s one.</p> <p>If <code>true</code>, emitted event is like below:</p> <pre><code>tag: syslog.unmatched # &#39;syslog&#39; depends on tag parameter time: parsed time record: {&quot;unmatched_line&quot;:&quot;broken line&quot;} </code></pre> <h3>buf_file: Add <code>path_suffix</code> parameter</h3> <p>You can change the suffix of buffer file. Here is an example:</p> <pre><code># default /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta # with &#39;path_suffix .buf&#39; /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf.meta </code></pre> <p>This parameter is useful when <code>.log</code> suffix is not fit for your system, </p> <h3>Bug fixes</h3> <ul> <li>http_server helper: Fix method re-define warning</li> </ul> <p>Enjoy logging!</p> 2019-07-29 <p>Hi users!</p> <p>We have released v1.6.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. These release includes several enhancements.</p> <h3>in_syslog: Add <code>emit_unmatched_lines</code> parameter</h3> <p>This parameter enables you to get unparseable log as an event. This is same as in_tail&#39;s one.</p> <p>If <code>true</code>, emitted event is like below:</p> <pre><code>tag: syslog.unmatched # &#39;syslog&#39; depends on tag parameter time: parsed time record: {&quot;unmatched_line&quot;:&quot;broken line&quot;} </code></pre> <h3>buf_file: Add <code>path_suffix</code> parameter</h3> <p>You can change the suffix of buffer file. Here is an example:</p> <pre><code># default /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta # with &#39;path_suffix .buf&#39; /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf /var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf.meta </code></pre> <p>This parameter is useful when <code>.log</code> suffix is not fit for your system, </p> <h3>Bug fixes</h3> <ul> <li>http_server helper: Fix method re-define warning</li> </ul> <p>Enjoy logging!</p>