Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v1.12.0 has been released https://www.fluentd.org/blog/fluentd-v1.12.0-has-been-released <p>Hi users!</p> <p>We have released v1.12.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3><code>in_tail</code>: Support <code>*</code> in <code>path</code> with log rotation</h3> <p>fluentd v1.12.0 resolves the limitation for <code>*</code> with log rotation. <code>follow_inodes true</code> enables the combination of <code>*</code> in path with log rotation inside same directory and <code>read_from_head true</code> without log duplication problem.</p> <pre><code>path /path/to/* read_from_head true follow_inodes true # without this parameter, log rotation may cause log duplication </code></pre> <h3><code>in_tail</code>: Support Linux Capability</h3> <p>Fluentd supports Linux capability via <code>capng_c</code> gem and <code>in_tail</code> now supports <code>CAP_DAC_READ_SEARCH</code>/<code>CAP_DAC_OVERRIDE</code> capabilities to read log files.</p> <p>See official article for more details: <a href="https://docs.fluentd.org/deployment/linux-capability">Linux Capability</a></p> <p>If you want more capability support in official plugins, file it on github.</p> <h3>Add <code>fluent-ctl</code> command</h3> <p><code>fluent-ctl</code> provides same interface to control fluentd process, <code>shutdown</code>/<code>restart</code>/<code>flush</code>/<code>reload</code>.</p> <ul> <li>On Linux, <code>fluent-ctl</code> uses Signals</li> <li>On Windows, <code>fluent-ctl</code> uses Windows Event</li> </ul> <p>Command example:</p> <pre><code># Usage: fluent-ctl action PID $ fluent-ctl shutdown 11111 </code></pre> <h3>Major bug fixes</h3> <ul> <li>output: Prevent <code>retry.step</code> from being called too many times in a short time</li> </ul> <p>Enjoy logging!</p> 2021-01-05 <p>Hi users!</p> <p>We have released v1.12.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3><code>in_tail</code>: Support <code>*</code> in <code>path</code> with log rotation</h3> <p>fluentd v1.12.0 resolves the limitation for <code>*</code> with log rotation. <code>follow_inodes true</code> enables the combination of <code>*</code> in path with log rotation inside same directory and <code>read_from_head true</code> without log duplication problem.</p> <pre><code>path /path/to/* read_from_head true follow_inodes true # without this parameter, log rotation may cause log duplication </code></pre> <h3><code>in_tail</code>: Support Linux Capability</h3> <p>Fluentd supports Linux capability via <code>capng_c</code> gem and <code>in_tail</code> now supports <code>CAP_DAC_READ_SEARCH</code>/<code>CAP_DAC_OVERRIDE</code> capabilities to read log files.</p> <p>See official article for more details: <a href="https://docs.fluentd.org/deployment/linux-capability">Linux Capability</a></p> <p>If you want more capability support in official plugins, file it on github.</p> <h3>Add <code>fluent-ctl</code> command</h3> <p><code>fluent-ctl</code> provides same interface to control fluentd process, <code>shutdown</code>/<code>restart</code>/<code>flush</code>/<code>reload</code>.</p> <ul> <li>On Linux, <code>fluent-ctl</code> uses Signals</li> <li>On Windows, <code>fluent-ctl</code> uses Windows Event</li> </ul> <p>Command example:</p> <pre><code># Usage: fluent-ctl action PID $ fluent-ctl shutdown 11111 </code></pre> <h3>Major bug fixes</h3> <ul> <li>output: Prevent <code>retry.step</code> from being called too many times in a short time</li> </ul> <p>Enjoy logging!</p> Fluentd v1.11.5 has been released https://www.fluentd.org/blog/fluentd-v1.11.5-has-been-released <p>Hi users!</p> <p>We have released v1.11.5. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>formatter: Provide <code>newline</code> parameter to support <code>CRLF</code></h3> <p>This parameter is for Windows environment. Users can choose <code>LF</code> or <code>CRLF</code> newline for <code>add_newline true</code> case.</p> <pre><code>newline lf # or crlf </code></pre> <h3>out_http: adding support for intermediate certificates</h3> <p>Add support for sending a client certificate chain for mutual TLS authentication in the HTTP output plugin.</p> <h3>Major bug fixes</h3> <ul> <li>Fix a bug that windows service isn&#39;t stopped gracefuly</li> </ul> <p>Enjoy logging!</p> 2020-11-06 <p>Hi users!</p> <p>We have released v1.11.5. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>formatter: Provide <code>newline</code> parameter to support <code>CRLF</code></h3> <p>This parameter is for Windows environment. Users can choose <code>LF</code> or <code>CRLF</code> newline for <code>add_newline true</code> case.</p> <pre><code>newline lf # or crlf </code></pre> <h3>out_http: adding support for intermediate certificates</h3> <p>Add support for sending a client certificate chain for mutual TLS authentication in the HTTP output plugin.</p> <h3>Major bug fixes</h3> <ul> <li>Fix a bug that windows service isn&#39;t stopped gracefuly</li> </ul> <p>Enjoy logging!</p> Fluentd Ecosystem Survey 2020 https://www.fluentd.org/blog/ecosystem-survey-2020 <p>Hey All,</p> <p>Thank you for being part of the Fluent Slack Community! As part of improving Fluentd and Fluent Bit, we want to know what pains and feedback you are experiencin to better manage our future together.</p> <p>We have created an anonymous survey that allows you to share this <a href="https://www.cognitoforms.com/Fluentecosystem/FluentEcosystemSurvey">here</a>.</p> <p><br/> <center><a href="https://www.cognitoforms.com/Fluentecosystem/FluentEcosystemSurvey"><button style="background-color:#0E83C8; color:#FFF; padding: 10px; ">Respond to the Survey</button></a></center> <br/></p> <p>Optionally, if you are interested you can add your email to give us direct feedback. Looking forward to your feedbacks!</p> 2020-10-28 <p>Hey All,</p> <p>Thank you for being part of the Fluent Slack Community! As part of improving Fluentd and Fluent Bit, we want to know what pains and feedback you are experiencin to better manage our future together.</p> <p>We have created an anonymous survey that allows you to share this <a href="https://www.cognitoforms.com/Fluentecosystem/FluentEcosystemSurvey">here</a>.</p> <p><br/> <center><a href="https://www.cognitoforms.com/Fluentecosystem/FluentEcosystemSurvey"><button style="background-color:#0E83C8; color:#FFF; padding: 10px; ">Respond to the Survey</button></a></center> <br/></p> <p>Optionally, if you are interested you can add your email to give us direct feedback. Looking forward to your feedbacks!</p> Fluentd v1.11.3 has been released https://www.fluentd.org/blog/fluentd-v1.11.3-has-been-released <p>Hi users!</p> <p>We have released v1.11.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>log: Add <code>ignore_same_log_interval</code> parameter</h3> <p>Fluentd has <code>ignore_repeated_log_interval</code> parameter but this is not enough for some frequent log generation case. For example, if the plugin generates several log messages in one action, logs are not repeated:</p> <pre><code># retry generates same error messages with ignore_repeated_log_interval def write(chunk) # process 1... log.error &quot;message1&quot; # process 2... log.error &quot;message2&quot; end </code></pre> <p><code>ignore_same_log_interval</code> resolves these cases. With this option, logger keeps to store all log messages for validation.</p> <pre><code>&lt;system&gt; ignore_same_log_interval 60s &lt;/system&gt; </code></pre> <h3>in<em>exec: Add `connect</em>mode` parameter to read stderr</h3> <p><code>in_exec</code> doesn&#39;t read stderr and this is not good for script debug. To resolve this problem, add <code>connect_mode</code> parameter to read stderr logs.</p> <pre><code>&lt;source&gt; @type exec tag somescript command python /path/to/script.py connect_mode read_with_stderr &lt;parse&gt; @type none &lt;/parse&gt; &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Fix calculation of timekey stats</li> <li>buffer: fix binmode usage for prevent gc</li> </ul> <p>Enjoy logging!</p> 2020-09-30 <p>Hi users!</p> <p>We have released v1.11.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>log: Add <code>ignore_same_log_interval</code> parameter</h3> <p>Fluentd has <code>ignore_repeated_log_interval</code> parameter but this is not enough for some frequent log generation case. For example, if the plugin generates several log messages in one action, logs are not repeated:</p> <pre><code># retry generates same error messages with ignore_repeated_log_interval def write(chunk) # process 1... log.error &quot;message1&quot; # process 2... log.error &quot;message2&quot; end </code></pre> <p><code>ignore_same_log_interval</code> resolves these cases. With this option, logger keeps to store all log messages for validation.</p> <pre><code>&lt;system&gt; ignore_same_log_interval 60s &lt;/system&gt; </code></pre> <h3>in<em>exec: Add `connect</em>mode` parameter to read stderr</h3> <p><code>in_exec</code> doesn&#39;t read stderr and this is not good for script debug. To resolve this problem, add <code>connect_mode</code> parameter to read stderr logs.</p> <pre><code>&lt;source&gt; @type exec tag somescript command python /path/to/script.py connect_mode read_with_stderr &lt;parse&gt; @type none &lt;/parse&gt; &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Fix calculation of timekey stats</li> <li>buffer: fix binmode usage for prevent gc</li> </ul> <p>Enjoy logging!</p> Fluentd v1.11.2 has been released https://www.fluentd.org/blog/fluentd-v1.11.2-has-been-released <p>Hi users!</p> <p>We have released v1.11.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>Allow regular expression in filter/match tag matching</h3> <p>Regular expression, <code>/pattern/</code>, is now allowed in tag matching:</p> <pre><code>&lt;source&gt; @type forward &lt;/source&gt; # For a.xxx tag &lt;filter a.**&gt; @type record_transformer &lt;record&gt; new_key a-started tag record &lt;/record&gt; &lt;/filter&gt; # For non a.xxx tag like b.xxx/c.xxx/... &lt;filter /(?!a\.).*/&gt; @type record_transformer &lt;record&gt; new_key other tag record &lt;/record&gt; &lt;/filter&gt; &lt;match **&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you send following events:</p> <pre><code>% echo &#39;{&quot;msg&quot;:&quot;hi&quot;}&#39; | fluent-cat a.foo % echo &#39;{&quot;msg&quot;:&quot;hi&quot;}&#39; | fluent-cat b.foo </code></pre> <p>the log shows:</p> <pre><code>2020-08-06 13:40:03.111196000 +0900 a.foo: {&quot;msg&quot;:&quot;hi&quot;,&quot;new_key&quot;:&quot;a-started tag record&quot;} 2020-08-06 13:40:09.492380000 +0900 b.foo: {&quot;msg&quot;:&quot;hi&quot;,&quot;new_key&quot;:&quot;other tag record&quot;} </code></pre> <p>Of course, simple data pipeline is important for robust log collection. If you need this feature, check your configuration/data flow first.</p> <h3><code>in_dummy</code> is renamed <code>in_sample</code></h3> <p>We recommend to use <code>sample</code> instead of <code>dummy</code>:</p> <pre><code>&lt;source&gt; @type sample sample {&quot;hello&quot;:&quot;world&quot;} &lt;/source&gt; # If you use fluentd v1.11.1 or earlier, use following configuration # Fluentd v2 will remove old configuration support &lt;source&gt; @type dummy dummy {&quot;hello&quot;:&quot;world&quot;} &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Fix log message for <code>chunk_limit_records</code> case</li> <li>buffer: Fix timekey optimization for non-windows platform</li> <li>cert: Raise an error for broken certificate file</li> <li>cert: Set TLS ciphers list correcty on older OpenSSL, e.g. openssl v1.0.2</li> </ul> <p>Enjoy logging!</p> 2020-08-06 <p>Hi users!</p> <p>We have released v1.11.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>Allow regular expression in filter/match tag matching</h3> <p>Regular expression, <code>/pattern/</code>, is now allowed in tag matching:</p> <pre><code>&lt;source&gt; @type forward &lt;/source&gt; # For a.xxx tag &lt;filter a.**&gt; @type record_transformer &lt;record&gt; new_key a-started tag record &lt;/record&gt; &lt;/filter&gt; # For non a.xxx tag like b.xxx/c.xxx/... &lt;filter /(?!a\.).*/&gt; @type record_transformer &lt;record&gt; new_key other tag record &lt;/record&gt; &lt;/filter&gt; &lt;match **&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you send following events:</p> <pre><code>% echo &#39;{&quot;msg&quot;:&quot;hi&quot;}&#39; | fluent-cat a.foo % echo &#39;{&quot;msg&quot;:&quot;hi&quot;}&#39; | fluent-cat b.foo </code></pre> <p>the log shows:</p> <pre><code>2020-08-06 13:40:03.111196000 +0900 a.foo: {&quot;msg&quot;:&quot;hi&quot;,&quot;new_key&quot;:&quot;a-started tag record&quot;} 2020-08-06 13:40:09.492380000 +0900 b.foo: {&quot;msg&quot;:&quot;hi&quot;,&quot;new_key&quot;:&quot;other tag record&quot;} </code></pre> <p>Of course, simple data pipeline is important for robust log collection. If you need this feature, check your configuration/data flow first.</p> <h3><code>in_dummy</code> is renamed <code>in_sample</code></h3> <p>We recommend to use <code>sample</code> instead of <code>dummy</code>:</p> <pre><code>&lt;source&gt; @type sample sample {&quot;hello&quot;:&quot;world&quot;} &lt;/source&gt; # If you use fluentd v1.11.1 or earlier, use following configuration # Fluentd v2 will remove old configuration support &lt;source&gt; @type dummy dummy {&quot;hello&quot;:&quot;world&quot;} &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Fix log message for <code>chunk_limit_records</code> case</li> <li>buffer: Fix timekey optimization for non-windows platform</li> <li>cert: Raise an error for broken certificate file</li> <li>cert: Set TLS ciphers list correcty on older OpenSSL, e.g. openssl v1.0.2</li> </ul> <p>Enjoy logging!</p>