Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v1.11.3 has been released https://www.fluentd.org/blog/fluentd-v1.11.3-has-been-released <p>Hi users!</p> <p>We have released v1.11.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>log: Add <code>ignore_same_log_interval</code> parameter</h3> <p>Fluentd has <code>ignore_repeated_log_interval</code> parameter but this is not enough for some frequent log generation case. For example, if the plugin generates several log messages in one action, logs are not repeated:</p> <pre><code># retry generates same error messages with ignore_repeated_log_interval def write(chunk) # process 1... log.error &quot;message1&quot; # process 2... log.error &quot;message2&quot; end </code></pre> <p><code>ignore_same_log_interval</code> resolves these cases. With this option, logger keeps to store all log messages for validation.</p> <pre><code>&lt;system&gt; ignore_same_log_interval 60s &lt;/system&gt; </code></pre> <h3>in<em>exec: Add `connect</em>mode` parameter to read stderr</h3> <p><code>in_exec</code> doesn&#39;t read stderr and this is not good for script debug. To resolve this problem, add <code>connect_mode</code> parameter to read stderr logs.</p> <pre><code>&lt;source&gt; @type exec tag somescript command python /path/to/script.py connect_mode read_with_stderr &lt;parse&gt; @type none &lt;/parse&gt; &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Fix calculation of timekey stats</li> <li>buffer: fix binmode usage for prevent gc</li> </ul> <p>Enjoy logging!</p> 2020-09-30 <p>Hi users!</p> <p>We have released v1.11.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>log: Add <code>ignore_same_log_interval</code> parameter</h3> <p>Fluentd has <code>ignore_repeated_log_interval</code> parameter but this is not enough for some frequent log generation case. For example, if the plugin generates several log messages in one action, logs are not repeated:</p> <pre><code># retry generates same error messages with ignore_repeated_log_interval def write(chunk) # process 1... log.error &quot;message1&quot; # process 2... log.error &quot;message2&quot; end </code></pre> <p><code>ignore_same_log_interval</code> resolves these cases. With this option, logger keeps to store all log messages for validation.</p> <pre><code>&lt;system&gt; ignore_same_log_interval 60s &lt;/system&gt; </code></pre> <h3>in<em>exec: Add `connect</em>mode` parameter to read stderr</h3> <p><code>in_exec</code> doesn&#39;t read stderr and this is not good for script debug. To resolve this problem, add <code>connect_mode</code> parameter to read stderr logs.</p> <pre><code>&lt;source&gt; @type exec tag somescript command python /path/to/script.py connect_mode read_with_stderr &lt;parse&gt; @type none &lt;/parse&gt; &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Fix calculation of timekey stats</li> <li>buffer: fix binmode usage for prevent gc</li> </ul> <p>Enjoy logging!</p> Fluentd v1.11.2 has been released https://www.fluentd.org/blog/fluentd-v1.11.2-has-been-released <p>Hi users!</p> <p>We have released v1.11.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>Allow regular expression in filter/match tag matching</h3> <p>Regular expression, <code>/pattern/</code>, is now allowed in tag matching:</p> <pre><code>&lt;source&gt; @type forward &lt;/source&gt; # For a.xxx tag &lt;filter a.**&gt; @type record_transformer &lt;record&gt; new_key a-started tag record &lt;/record&gt; &lt;/filter&gt; # For non a.xxx tag like b.xxx/c.xxx/... &lt;filter /(?!a\.).*/&gt; @type record_transformer &lt;record&gt; new_key other tag record &lt;/record&gt; &lt;/filter&gt; &lt;match **&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you send following events:</p> <pre><code>% echo &#39;{&quot;msg&quot;:&quot;hi&quot;}&#39; | fluent-cat a.foo % echo &#39;{&quot;msg&quot;:&quot;hi&quot;}&#39; | fluent-cat b.foo </code></pre> <p>the log shows:</p> <pre><code>2020-08-06 13:40:03.111196000 +0900 a.foo: {&quot;msg&quot;:&quot;hi&quot;,&quot;new_key&quot;:&quot;a-started tag record&quot;} 2020-08-06 13:40:09.492380000 +0900 b.foo: {&quot;msg&quot;:&quot;hi&quot;,&quot;new_key&quot;:&quot;other tag record&quot;} </code></pre> <p>Of course, simple data pipeline is important for robust log collection. If you need this feature, check your configuration/data flow first.</p> <h3><code>in_dummy</code> is renamed <code>in_sample</code></h3> <p>We recommend to use <code>sample</code> instead of <code>dummy</code>:</p> <pre><code>&lt;source&gt; @type sample sample {&quot;hello&quot;:&quot;world&quot;} &lt;/source&gt; # If you use fluentd v1.11.1 or earlier, use following configuration # Fluentd v2 will remove old configuration support &lt;source&gt; @type dummy dummy {&quot;hello&quot;:&quot;world&quot;} &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Fix log message for <code>chunk_limit_records</code> case</li> <li>buffer: Fix timekey optimization for non-windows platform</li> <li>cert: Raise an error for broken certificate file</li> <li>cert: Set TLS ciphers list correcty on older OpenSSL, e.g. openssl v1.0.2</li> </ul> <p>Enjoy logging!</p> 2020-08-06 <p>Hi users!</p> <p>We have released v1.11.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>Allow regular expression in filter/match tag matching</h3> <p>Regular expression, <code>/pattern/</code>, is now allowed in tag matching:</p> <pre><code>&lt;source&gt; @type forward &lt;/source&gt; # For a.xxx tag &lt;filter a.**&gt; @type record_transformer &lt;record&gt; new_key a-started tag record &lt;/record&gt; &lt;/filter&gt; # For non a.xxx tag like b.xxx/c.xxx/... &lt;filter /(?!a\.).*/&gt; @type record_transformer &lt;record&gt; new_key other tag record &lt;/record&gt; &lt;/filter&gt; &lt;match **&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you send following events:</p> <pre><code>% echo &#39;{&quot;msg&quot;:&quot;hi&quot;}&#39; | fluent-cat a.foo % echo &#39;{&quot;msg&quot;:&quot;hi&quot;}&#39; | fluent-cat b.foo </code></pre> <p>the log shows:</p> <pre><code>2020-08-06 13:40:03.111196000 +0900 a.foo: {&quot;msg&quot;:&quot;hi&quot;,&quot;new_key&quot;:&quot;a-started tag record&quot;} 2020-08-06 13:40:09.492380000 +0900 b.foo: {&quot;msg&quot;:&quot;hi&quot;,&quot;new_key&quot;:&quot;other tag record&quot;} </code></pre> <p>Of course, simple data pipeline is important for robust log collection. If you need this feature, check your configuration/data flow first.</p> <h3><code>in_dummy</code> is renamed <code>in_sample</code></h3> <p>We recommend to use <code>sample</code> instead of <code>dummy</code>:</p> <pre><code>&lt;source&gt; @type sample sample {&quot;hello&quot;:&quot;world&quot;} &lt;/source&gt; # If you use fluentd v1.11.1 or earlier, use following configuration # Fluentd v2 will remove old configuration support &lt;source&gt; @type dummy dummy {&quot;hello&quot;:&quot;world&quot;} &lt;/source&gt; </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Fix log message for <code>chunk_limit_records</code> case</li> <li>buffer: Fix timekey optimization for non-windows platform</li> <li>cert: Raise an error for broken certificate file</li> <li>cert: Set TLS ciphers list correcty on older OpenSSL, e.g. openssl v1.0.2</li> </ul> <p>Enjoy logging!</p> Fluentd v1.11.1 has been released https://www.fluentd.org/blog/fluentd-v1.11.1-has-been-released <p>Hi users!</p> <p>We have released v1.11.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>in_http: Improve time field handling</h3> <p><code>in_http</code> now supports time parsing in record field for default json/msgpack request. Here is configuration example:</p> <pre><code>&lt;source&gt; @type http @id input_http port 8888 &lt;parse&gt; time_format %iso8601 time_key logtime keep_time_key true &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you post following requests:</p> <pre><code># ruby code example require &#39;net/http&#39; require &#39;json&#39; record = {&#39;json&#39; =&gt; {&#39;k&#39; =&gt; &#39;hello1&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:27+09:00&#39;}.to_json} records = {&#39;json&#39; =&gt; [ {&#39;k&#39; =&gt; &#39;hello2&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:28+09:00&#39;}, {&#39;k&#39; =&gt; &#39;hello3&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:29+09:00&#39;} ].to_json} def post(path, params) http = Net::HTTP.new(&#39;127.0.0.1&#39;, 8888) req = Net::HTTP::Post.new(path, {}) req.set_form_data(params) http.request(req) end post(&quot;/test.http&quot;, record) post(&quot;/test.http&quot;, records) </code></pre> <p>the log shows expected result:</p> <pre><code>2020-06-10 11:14:27.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello1&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:27+09:00&quot;} 2020-06-10 11:14:28.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello2&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:28+09:00&quot;} 2020-06-10 11:14:29.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello3&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:29+09:00&quot;} </code></pre> <p>Before, this configuration doesn&#39;t work because <code>in_http</code> assumes the time value is floating point.</p> <h3>Major bug fixes</h3> <ul> <li>in_tail: Use actual path instead of base pattern for ignore list</li> <li>child_process helper: Fix child process failure due to SIGPIPE if the command uses stdout</li> </ul> <p>Enjoy logging!</p> 2020-06-22 <p>Hi users!</p> <p>We have released v1.11.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>in_http: Improve time field handling</h3> <p><code>in_http</code> now supports time parsing in record field for default json/msgpack request. Here is configuration example:</p> <pre><code>&lt;source&gt; @type http @id input_http port 8888 &lt;parse&gt; time_format %iso8601 time_key logtime keep_time_key true &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you post following requests:</p> <pre><code># ruby code example require &#39;net/http&#39; require &#39;json&#39; record = {&#39;json&#39; =&gt; {&#39;k&#39; =&gt; &#39;hello1&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:27+09:00&#39;}.to_json} records = {&#39;json&#39; =&gt; [ {&#39;k&#39; =&gt; &#39;hello2&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:28+09:00&#39;}, {&#39;k&#39; =&gt; &#39;hello3&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:29+09:00&#39;} ].to_json} def post(path, params) http = Net::HTTP.new(&#39;127.0.0.1&#39;, 8888) req = Net::HTTP::Post.new(path, {}) req.set_form_data(params) http.request(req) end post(&quot;/test.http&quot;, record) post(&quot;/test.http&quot;, records) </code></pre> <p>the log shows expected result:</p> <pre><code>2020-06-10 11:14:27.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello1&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:27+09:00&quot;} 2020-06-10 11:14:28.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello2&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:28+09:00&quot;} 2020-06-10 11:14:29.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello3&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:29+09:00&quot;} </code></pre> <p>Before, this configuration doesn&#39;t work because <code>in_http</code> assumes the time value is floating point.</p> <h3>Major bug fixes</h3> <ul> <li>in_tail: Use actual path instead of base pattern for ignore list</li> <li>child_process helper: Fix child process failure due to SIGPIPE if the command uses stdout</li> </ul> <p>Enjoy logging!</p> Fluentd v1.11.0 has been released https://www.fluentd.org/blog/fluentd-v1.11.0-has-been-released <p>Hi users!</p> <p>We have released v1.11.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>in_unix: Use v1 API</h3> <p>Migrate to v1 plugin API.</p> <p>In addition, <code>in_unix</code> now supports <code>tag</code> parameter to use fixed tag.</p> <h3>Improve syslog parser</h3> <p>There are several improvements:</p> <ul> <li>Improve <code>message_format auto</code> performance by avoiding object allocation</li> <li>Support any <code>time_format</code> for RFC3164 with <code>parser_type string</code> </li> <li>Support <code>parser_type string</code> for RFC5424. This parser is 2x faster than <code>parser_type regexp</code></li> </ul> <p>We now recommend to use <code>parser_type string</code> for both RFC3164 and RFC5424.</p> <h3>Major bug fixes</h3> <ul> <li>in_gc_stat: Add <code>use_symbol_keys</code> parameter to support non-stdout plugin.</li> </ul> <p>Enjoy logging!</p> 2020-06-04 <p>Hi users!</p> <p>We have released v1.11.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>in_unix: Use v1 API</h3> <p>Migrate to v1 plugin API.</p> <p>In addition, <code>in_unix</code> now supports <code>tag</code> parameter to use fixed tag.</p> <h3>Improve syslog parser</h3> <p>There are several improvements:</p> <ul> <li>Improve <code>message_format auto</code> performance by avoiding object allocation</li> <li>Support any <code>time_format</code> for RFC3164 with <code>parser_type string</code> </li> <li>Support <code>parser_type string</code> for RFC5424. This parser is 2x faster than <code>parser_type regexp</code></li> </ul> <p>We now recommend to use <code>parser_type string</code> for both RFC3164 and RFC5424.</p> <h3>Major bug fixes</h3> <ul> <li>in_gc_stat: Add <code>use_symbol_keys</code> parameter to support non-stdout plugin.</li> </ul> <p>Enjoy logging!</p> Fluentd v1.10.4 has been released https://www.fluentd.org/blog/fluentd-v1.10.4-has-been-released <p>Hi users!</p> <p>We have released v1.10.4. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release fixes several bugs.</p> <h3>out_http: Support single json array payload</h3> <p><code>out_http</code> uses <code>ndjson</code> for json payload by default.</p> <pre><code>{&quot;key&quot;:&quot;value1&quot;} {&quot;key&quot;:&quot;value2&quot;} ... {&quot;key&quot;:&quot;valueN&quot;} </code></pre> <p>You can change payload format to single json array by setting <code>json_array true</code>.</p> <pre><code># conf &lt;match pattern&gt; @type http # other parameters... json_array true &lt;format&gt; @type json add_newline false &lt;/format&gt; &lt;/match&gt; # payload [{&quot;key&quot;:&quot;value1&quot;},{&quot;key&quot;:&quot;value2&quot;},...,{&quot;key&quot;:&quot;valueN&quot;}] </code></pre> <h3>Major bug fixes</h3> <ul> <li>supervisor: Call <code>File.umask(0)</code> for standalone worker</li> <li>out_forward: Fix <code>ZeroDivisionError</code> issue with <code>weight 0</code></li> </ul> <p>Enjoy logging!</p> 2020-05-12 <p>Hi users!</p> <p>We have released v1.10.4. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release fixes several bugs.</p> <h3>out_http: Support single json array payload</h3> <p><code>out_http</code> uses <code>ndjson</code> for json payload by default.</p> <pre><code>{&quot;key&quot;:&quot;value1&quot;} {&quot;key&quot;:&quot;value2&quot;} ... {&quot;key&quot;:&quot;valueN&quot;} </code></pre> <p>You can change payload format to single json array by setting <code>json_array true</code>.</p> <pre><code># conf &lt;match pattern&gt; @type http # other parameters... json_array true &lt;format&gt; @type json add_newline false &lt;/format&gt; &lt;/match&gt; # payload [{&quot;key&quot;:&quot;value1&quot;},{&quot;key&quot;:&quot;value2&quot;},...,{&quot;key&quot;:&quot;valueN&quot;}] </code></pre> <h3>Major bug fixes</h3> <ul> <li>supervisor: Call <code>File.umask(0)</code> for standalone worker</li> <li>out_forward: Fix <code>ZeroDivisionError</code> issue with <code>weight 0</code></li> </ul> <p>Enjoy logging!</p>