Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v1.2.4 has been released https://www.fluentd.org/blog/fluentd-v1.2.4-has-been-released <p>Hi users!</p> <p>We have released v1.2.4. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is for bug fixes.</p> <h3>Major bug fixes</h3> <ul> <li><p>output: Consider timezone with larger timekey. This fixes unexpected buffer flush with <code>1d</code> timekey.</p></li> <li><p>server helper: Fix connection leak by close timing issue</p></li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p> 2018-08-01 <p>Hi users!</p> <p>We have released v1.2.4. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is for bug fixes.</p> <h3>Major bug fixes</h3> <ul> <li><p>output: Consider timezone with larger timekey. This fixes unexpected buffer flush with <code>1d</code> timekey.</p></li> <li><p>server helper: Fix connection leak by close timing issue</p></li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p> Fluentd v1.2.2 has been released https://www.fluentd.org/blog/fluentd-v1.2.2-has-been-released <p>Hi users!</p> <p>We have released v1.2.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is mainly for bug fixes.</p> <h2>filter_parser: Add <code>remove_key_name_field</code> parameter</h2> <p>With <code>reserve_data true</code>, <code>key_name</code> field is kept in the record.</p> <p>Original record:</p> <pre><code>{&quot;k1&quot;:&quot;v1&quot;,&quot;log&quot;:&quot;{\&quot;k2\&quot;:\&quot;v2\&quot;}&quot;} </code></pre> <p>Parsed result:</p> <pre><code>{&quot;k1&quot;:&quot;v1&quot;,&quot;log&quot;:&quot;{\&quot;k2\&quot;:\&quot;v2\&quot;}&quot;,&quot;k2&quot;:&quot;v2&quot;} </code></pre> <p>But we often don&#39;t need original <code>key_name</code> field when parsing is succeeded. You can remove <code>key_name</code> field with <code>remove_key_name_field true</code>. Parsed result is below:</p> <pre><code>{&quot;k1&quot;:&quot;v1&quot;,&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Wait for all chunks being purged before deleting <code>@queued_num</code> items</li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p> 2018-06-12 <p>Hi users!</p> <p>We have released v1.2.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is mainly for bug fixes.</p> <h2>filter_parser: Add <code>remove_key_name_field</code> parameter</h2> <p>With <code>reserve_data true</code>, <code>key_name</code> field is kept in the record.</p> <p>Original record:</p> <pre><code>{&quot;k1&quot;:&quot;v1&quot;,&quot;log&quot;:&quot;{\&quot;k2\&quot;:\&quot;v2\&quot;}&quot;} </code></pre> <p>Parsed result:</p> <pre><code>{&quot;k1&quot;:&quot;v1&quot;,&quot;log&quot;:&quot;{\&quot;k2\&quot;:\&quot;v2\&quot;}&quot;,&quot;k2&quot;:&quot;v2&quot;} </code></pre> <p>But we often don&#39;t need original <code>key_name</code> field when parsing is succeeded. You can remove <code>key_name</code> field with <code>remove_key_name_field true</code>. Parsed result is below:</p> <pre><code>{&quot;k1&quot;:&quot;v1&quot;,&quot;k2&quot;:&quot;v2&quot;} </code></pre> <h3>Major bug fixes</h3> <ul> <li>buffer: Wait for all chunks being purged before deleting <code>@queued_num</code> items</li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p> Fluentd v1.2.1 has been released https://www.fluentd.org/blog/fluentd-v1.2.1-has-been-released <p>Hi users!</p> <p>We have released v1.2.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is mainly for bug fixes.</p> <h2>Counter API: Add <code>wait</code> to Client</h2> <p>This is handy API for error check. Using <code>get</code>, you need to check error response from the response body. <code>wait</code> raises an exception when API call has a problem.</p> <pre><code>require &#39;fluent/plugin/filter&#39; module Fluent module Plugin class CounterFilter &lt; Filter Plugin.register_filter(&#39;counter&#39;, self) helpers :counter def start super @client = counter_client_create(scope: &#39;test&#39;) @client.establish(&#39;counter&#39;) # if init call returns an error, exception is raised begin @client.init(:name =&gt; &#39;num&#39;, type: &#39;numeric&#39;, reset_interval: 10).wait rescue Fluent::Counter::BaseError # process client specific error end end # ... end end end </code></pre> <h3>Major bug fixes</h3> <ul> <li>in<em>tcp/in</em>udp: Fix source<em>hostname</em>key to set hostname correctly</li> <li>out_file: Temporal fix for broken gzipped files with <code>gzip</code> and <code>append true</code></li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p> 2018-05-23 <p>Hi users!</p> <p>We have released v1.2.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is mainly for bug fixes.</p> <h2>Counter API: Add <code>wait</code> to Client</h2> <p>This is handy API for error check. Using <code>get</code>, you need to check error response from the response body. <code>wait</code> raises an exception when API call has a problem.</p> <pre><code>require &#39;fluent/plugin/filter&#39; module Fluent module Plugin class CounterFilter &lt; Filter Plugin.register_filter(&#39;counter&#39;, self) helpers :counter def start super @client = counter_client_create(scope: &#39;test&#39;) @client.establish(&#39;counter&#39;) # if init call returns an error, exception is raised begin @client.init(:name =&gt; &#39;num&#39;, type: &#39;numeric&#39;, reset_interval: 10).wait rescue Fluent::Counter::BaseError # process client specific error end end # ... end end end </code></pre> <h3>Major bug fixes</h3> <ul> <li>in<em>tcp/in</em>udp: Fix source<em>hostname</em>key to set hostname correctly</li> <li>out_file: Temporal fix for broken gzipped files with <code>gzip</code> and <code>append true</code></li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p> Fluentd v1.2.0 has been released https://www.fluentd.org/blog/fluentd-v1.2.0-has-been-released <p>Hi users!</p> <p>We have released v1.2.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new features and improvements.</p> <h2>output: Backup for broken chunks</h2> <p>Fluentd receives various events from various data sources. This sometimes have a problem in Output plugins.</p> <p>For example, if one application generates invalid events for data destination, e.g. schema mismatch, buffer flush always failed. Other case is generated events are invalid for output configuration, e.g. required field is missing.</p> <p>Fluentd has retry feature for temporal failures but there errors are never succeeded. So Fluentd should not retry unexpected &quot;broken chunks&quot;.</p> <p>Since v1.2.0, fluentd routes broken chunks to backup directory. By default, backup root directory is <code>/tmp/fluent</code>. If you set <code>root_dir</code> in <code>&lt;system&gt;</code>, <code>root_dir</code> is used. File path consists of several parameters for unique path and <code>${root_dir}/backup/worker${worker_id}/${plugin_id}/{chunk_id}.log</code> is path template.</p> <p>If you have following configuration and unrecoverable error happens inside <code>@type sql</code> plugin,</p> <pre><code>&lt;system&gt; root_dir /var/log/fluentd &lt;/system&gt; # ... &lt;match app.**&gt; @type sql @id out_sql &lt;buffer&gt; # ... &lt;/buffer&gt; &lt;/match&gt; </code></pre> <p>chunk is routed to <code>/var/log/fluentd/backup/worker0/out_sql/56644156d63a108eda2f487c58140736.log</code>.</p> <p>Currently, fluentd routes chunks to backup directory when Output plugin raises following errors during buffer flush.</p> <ul> <li><code>Fluent::UnrecoverableError</code>: Output plugin can raise this error to avoid retry for plugin specific case.</li> <li><code>TypeError</code>: This error sometimes happens when an event has unexpected type in target field.</li> <li><code>ArgumentError</code>: This error sometimes happens when library usage is wrong in plugins.</li> <li><code>NoMethodError</code>: This error sometimes happens when events and configuration are mismatched.</li> </ul> <p>Fluentd continues to retry buffer flush when other error happens.</p> <h2>New API: Counter API</h2> <p>This is for plugin developers. Counter API consists of server and client. Server stores counter values and client calls APIs to store values to server.</p> <p>Here is example configuration:</p> <pre><code>&lt;system&gt; &lt;counter_server&gt; scope test bind 127.0.0.1 port 25000 backup_path /tmp/counter_backup.json &lt;/counter_server&gt; &lt;counter_client&gt; host 127.0.0.1 port 25000 &lt;/counter_client&gt; &lt;/system&gt; </code></pre> <p>And filter implementation example:</p> <pre><code>require &#39;fluent/plugin/filter&#39; module Fluent module Plugin class CounterFilter &lt; Filter Plugin.register_filter(&#39;counter&#39;, self) helpers :counter def start super @client = counter_client_create(scope: &#39;test&#39;) # scope value is same with `&lt;counter_server&gt;` @client.establish(&#39;counter&#39;).get @client.init(:name =&gt; &#39;num&#39;, type: &#39;numeric&#39;, reset_interval: 10).get end def filter(tag, time, record) @client.inc(:name =&gt; &#39;num&#39;, :value =&gt; 5) p @client.get(&#39;num&#39;).data.first[&quot;current&quot;] # Show current value record end end end end </code></pre> <p>This API is useful for storing metrics under multiprocess environment. We will add API documents.</p> <h3>filter_grep: Support for <code>&lt;and&gt;</code> and <code>&lt;or&gt;</code> sections</h3> <p><code>grep</code> filter now supports <code>and</code> / <code>or</code> condition for <code>&lt;regexp&gt;</code> / <code>&lt;exclude&gt;</code> Before v1.2.0, <code>grep</code> filter supports only <code>and</code> <code>&lt;regexp&gt;</code> and <code>or</code> <code>&lt;exclude&gt;</code> patterns.</p> <pre><code>&lt;filter pattern&gt; # These &lt;regexp&gt;s are &quot;and&quot; &lt;regexp&gt; key level pattern ^ERROR|WARN$ &lt;/regexp&gt; &lt;regexp&gt; key method pattern ^GET|POST$ &lt;/regexp&gt; # These &lt;exclude&gt;s are &quot;or&quot; &lt;exclude&gt; key level pattern ^WARN$ &lt;/exclude&gt; &lt;exclude&gt; key method pattern ^GET$ &lt;/exclude&gt; &lt;/filter&gt; </code></pre> <p>v1.2.0 adds <code>&lt;and&gt;</code> and <code>&lt;or&gt;</code> sections to support more patterns.</p> <p>Here is configuration example:</p> <pre><code>&lt;filter pattern&gt; &lt;or&gt; &lt;regexp&gt; key level pattern ^ERROR|WARN$ &lt;/regexp&gt; &lt;regexp&gt; key method pattern ^GET|POST$ &lt;/regexp&gt; &lt;/or&gt; &lt;and&gt; &lt;exclude&gt; key level pattern ^WARN$ &lt;/exclude&gt; &lt;exclude&gt; key method pattern ^GET$ &lt;/exclude&gt; &lt;/and&gt; &lt;/filter&gt; </code></pre> <p>If you pass this data streams:</p> <pre><code>{&quot;time&quot; : &quot;2013/01/13T07:02:11.124202&quot;, &quot;level&quot; : &quot;INFO&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/ping&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:13.232645&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/auth&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:21.542145&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/favicon.ico&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:43.632145&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/login&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:44.959307&quot;, &quot;level&quot; : &quot;ERROR&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/login&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:45.444992&quot;, &quot;level&quot; : &quot;ERROR&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/ping&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:51.247941&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/info&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:53.108366&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/ban&quot;} </code></pre> <p>filtered result is below:</p> <pre><code>{&quot;time&quot; : &quot;2013/01/13T07:02:11.124202&quot;, &quot;level&quot; : &quot;INFO&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/ping&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:13.232645&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/auth&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:43.632145&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/login&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:44.959307&quot;, &quot;level&quot; : &quot;ERROR&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/login&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:45.444992&quot;, &quot;level&quot; : &quot;ERROR&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/ping&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:53.108366&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/ban&quot;} </code></pre> <h3>Major bug fixes</h3> <ul> <li>server helper: Close invalid socket when ssl error happens on reading</li> <li>log: Fix unexpected implementation bug when log rotation setting is applied</li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p> 2018-05-05 <p>Hi users!</p> <p>We have released v1.2.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes new features and improvements.</p> <h2>output: Backup for broken chunks</h2> <p>Fluentd receives various events from various data sources. This sometimes have a problem in Output plugins.</p> <p>For example, if one application generates invalid events for data destination, e.g. schema mismatch, buffer flush always failed. Other case is generated events are invalid for output configuration, e.g. required field is missing.</p> <p>Fluentd has retry feature for temporal failures but there errors are never succeeded. So Fluentd should not retry unexpected &quot;broken chunks&quot;.</p> <p>Since v1.2.0, fluentd routes broken chunks to backup directory. By default, backup root directory is <code>/tmp/fluent</code>. If you set <code>root_dir</code> in <code>&lt;system&gt;</code>, <code>root_dir</code> is used. File path consists of several parameters for unique path and <code>${root_dir}/backup/worker${worker_id}/${plugin_id}/{chunk_id}.log</code> is path template.</p> <p>If you have following configuration and unrecoverable error happens inside <code>@type sql</code> plugin,</p> <pre><code>&lt;system&gt; root_dir /var/log/fluentd &lt;/system&gt; # ... &lt;match app.**&gt; @type sql @id out_sql &lt;buffer&gt; # ... &lt;/buffer&gt; &lt;/match&gt; </code></pre> <p>chunk is routed to <code>/var/log/fluentd/backup/worker0/out_sql/56644156d63a108eda2f487c58140736.log</code>.</p> <p>Currently, fluentd routes chunks to backup directory when Output plugin raises following errors during buffer flush.</p> <ul> <li><code>Fluent::UnrecoverableError</code>: Output plugin can raise this error to avoid retry for plugin specific case.</li> <li><code>TypeError</code>: This error sometimes happens when an event has unexpected type in target field.</li> <li><code>ArgumentError</code>: This error sometimes happens when library usage is wrong in plugins.</li> <li><code>NoMethodError</code>: This error sometimes happens when events and configuration are mismatched.</li> </ul> <p>Fluentd continues to retry buffer flush when other error happens.</p> <h2>New API: Counter API</h2> <p>This is for plugin developers. Counter API consists of server and client. Server stores counter values and client calls APIs to store values to server.</p> <p>Here is example configuration:</p> <pre><code>&lt;system&gt; &lt;counter_server&gt; scope test bind 127.0.0.1 port 25000 backup_path /tmp/counter_backup.json &lt;/counter_server&gt; &lt;counter_client&gt; host 127.0.0.1 port 25000 &lt;/counter_client&gt; &lt;/system&gt; </code></pre> <p>And filter implementation example:</p> <pre><code>require &#39;fluent/plugin/filter&#39; module Fluent module Plugin class CounterFilter &lt; Filter Plugin.register_filter(&#39;counter&#39;, self) helpers :counter def start super @client = counter_client_create(scope: &#39;test&#39;) # scope value is same with `&lt;counter_server&gt;` @client.establish(&#39;counter&#39;).get @client.init(:name =&gt; &#39;num&#39;, type: &#39;numeric&#39;, reset_interval: 10).get end def filter(tag, time, record) @client.inc(:name =&gt; &#39;num&#39;, :value =&gt; 5) p @client.get(&#39;num&#39;).data.first[&quot;current&quot;] # Show current value record end end end end </code></pre> <p>This API is useful for storing metrics under multiprocess environment. We will add API documents.</p> <h3>filter_grep: Support for <code>&lt;and&gt;</code> and <code>&lt;or&gt;</code> sections</h3> <p><code>grep</code> filter now supports <code>and</code> / <code>or</code> condition for <code>&lt;regexp&gt;</code> / <code>&lt;exclude&gt;</code> Before v1.2.0, <code>grep</code> filter supports only <code>and</code> <code>&lt;regexp&gt;</code> and <code>or</code> <code>&lt;exclude&gt;</code> patterns.</p> <pre><code>&lt;filter pattern&gt; # These &lt;regexp&gt;s are &quot;and&quot; &lt;regexp&gt; key level pattern ^ERROR|WARN$ &lt;/regexp&gt; &lt;regexp&gt; key method pattern ^GET|POST$ &lt;/regexp&gt; # These &lt;exclude&gt;s are &quot;or&quot; &lt;exclude&gt; key level pattern ^WARN$ &lt;/exclude&gt; &lt;exclude&gt; key method pattern ^GET$ &lt;/exclude&gt; &lt;/filter&gt; </code></pre> <p>v1.2.0 adds <code>&lt;and&gt;</code> and <code>&lt;or&gt;</code> sections to support more patterns.</p> <p>Here is configuration example:</p> <pre><code>&lt;filter pattern&gt; &lt;or&gt; &lt;regexp&gt; key level pattern ^ERROR|WARN$ &lt;/regexp&gt; &lt;regexp&gt; key method pattern ^GET|POST$ &lt;/regexp&gt; &lt;/or&gt; &lt;and&gt; &lt;exclude&gt; key level pattern ^WARN$ &lt;/exclude&gt; &lt;exclude&gt; key method pattern ^GET$ &lt;/exclude&gt; &lt;/and&gt; &lt;/filter&gt; </code></pre> <p>If you pass this data streams:</p> <pre><code>{&quot;time&quot; : &quot;2013/01/13T07:02:11.124202&quot;, &quot;level&quot; : &quot;INFO&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/ping&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:13.232645&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/auth&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:21.542145&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/favicon.ico&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:43.632145&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/login&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:44.959307&quot;, &quot;level&quot; : &quot;ERROR&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/login&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:45.444992&quot;, &quot;level&quot; : &quot;ERROR&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/ping&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:51.247941&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/info&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:53.108366&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/ban&quot;} </code></pre> <p>filtered result is below:</p> <pre><code>{&quot;time&quot; : &quot;2013/01/13T07:02:11.124202&quot;, &quot;level&quot; : &quot;INFO&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/ping&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:13.232645&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/auth&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:43.632145&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/login&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:44.959307&quot;, &quot;level&quot; : &quot;ERROR&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/login&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:45.444992&quot;, &quot;level&quot; : &quot;ERROR&quot;, &quot;method&quot; : &quot;GET&quot;, &quot;path&quot; : &quot;/ping&quot;} {&quot;time&quot; : &quot;2013/01/13T07:02:53.108366&quot;, &quot;level&quot; : &quot;WARN&quot;, &quot;method&quot; : &quot;POST&quot;, &quot;path&quot; : &quot;/ban&quot;} </code></pre> <h3>Major bug fixes</h3> <ul> <li>server helper: Close invalid socket when ssl error happens on reading</li> <li>log: Fix unexpected implementation bug when log rotation setting is applied</li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p> Fluentd v1.1.3 has been released https://www.fluentd.org/blog/fluentd-v1.1.3-has-been-released <p>Hi users!</p> <p>We have released v1.1.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements and bug fixes.</p> <h3>output: Support negative array index for tag placeholder</h3> <p>We can use array index for tag placeholder. This is useful for accessing tag parts.</p> <pre><code>&lt;match app.**&gt; @type foo param value-${tag[1]} # if tag is &#39;app.foo.bar&#39;, ${tag[1]} is &#39;foo&#39; &lt;/source&gt; </code></pre> <p>Since v1.1.3, you can also use negative array index for tag placeholder. The behaviour is same as ruby&#39;s negative array index.</p> <pre><code>&lt;match app.**&gt; @type foo param value-${tag[-1]} # if tag is &#39;app.foo.bar&#39;, ${tag[-1]} is &#39;bar&#39; &lt;/source&gt; </code></pre> <h3>buffer: Add <code>queued_chunks_limit_size</code> to control the number of queued chunks</h3> <p>This new <code>queued_chunks_limit_size</code> parameter mitigates lots of queued chunks issue with frequent enqueuing.</p> <p>Sometimes users set smaller <code>flush_interval</code>, e.g. <code>1s</code>, for log forwarding. This is no problem on healthy environment. But if the destination is slower or unstable, output&#39;s flush fails and retry is started. In such situation, lots of small queued chunks are generated in the buffer and it consumes lots of fd resources when you use <code>file</code> buffer. To avoid this problem, <code>queued_chunks_limit_size</code> is useful. If you set <code>queued_chunks_limit_size 5</code>, staged chunks are not enqueued until the number of waiting enqueued chunks is less than <code>5</code>.</p> <p>Note that this check is for interval based enqueuing for now. It means if your staged chunk reaches <code>chunk_limit_size</code>, its chunks is enqueued even if the number of waiting enqueued chunks is greater than <code>queued_chunks_limit_size</code>.</p> <h3>Major bug fixes</h3> <ul> <li>output: Delete empty queued_num field after purging chunks. This fixes memory leak when chunk keys include <code>time</code></li> <li>out_forward: The node should be disabled when TLS socket for ack returns an error</li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p> 2018-04-05 <p>Hi users!</p> <p>We have released v1.1.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements and bug fixes.</p> <h3>output: Support negative array index for tag placeholder</h3> <p>We can use array index for tag placeholder. This is useful for accessing tag parts.</p> <pre><code>&lt;match app.**&gt; @type foo param value-${tag[1]} # if tag is &#39;app.foo.bar&#39;, ${tag[1]} is &#39;foo&#39; &lt;/source&gt; </code></pre> <p>Since v1.1.3, you can also use negative array index for tag placeholder. The behaviour is same as ruby&#39;s negative array index.</p> <pre><code>&lt;match app.**&gt; @type foo param value-${tag[-1]} # if tag is &#39;app.foo.bar&#39;, ${tag[-1]} is &#39;bar&#39; &lt;/source&gt; </code></pre> <h3>buffer: Add <code>queued_chunks_limit_size</code> to control the number of queued chunks</h3> <p>This new <code>queued_chunks_limit_size</code> parameter mitigates lots of queued chunks issue with frequent enqueuing.</p> <p>Sometimes users set smaller <code>flush_interval</code>, e.g. <code>1s</code>, for log forwarding. This is no problem on healthy environment. But if the destination is slower or unstable, output&#39;s flush fails and retry is started. In such situation, lots of small queued chunks are generated in the buffer and it consumes lots of fd resources when you use <code>file</code> buffer. To avoid this problem, <code>queued_chunks_limit_size</code> is useful. If you set <code>queued_chunks_limit_size 5</code>, staged chunks are not enqueued until the number of waiting enqueued chunks is less than <code>5</code>.</p> <p>Note that this check is for interval based enqueuing for now. It means if your staged chunk reaches <code>chunk_limit_size</code>, its chunks is enqueued even if the number of waiting enqueued chunks is greater than <code>queued_chunks_limit_size</code>.</p> <h3>Major bug fixes</h3> <ul> <li>output: Delete empty queued_num field after purging chunks. This fixes memory leak when chunk keys include <code>time</code></li> <li>out_forward: The node should be disabled when TLS socket for ack returns an error</li> </ul> <p>Thanks for submitting bug reports and patches :)</p> <p>Enjoy logging!</p>