Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd v1.11.1 has been released https://www.fluentd.org/blog/fluentd-v1.11.1-has-been-released <p>Hi users!</p> <p>We have released v1.11.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>in_http: Improve time field handling</h3> <p><code>in_http</code> now supports time parsing in record field for default json/msgpack request. Here is configuration example:</p> <pre><code>&lt;source&gt; @type http @id input_http port 8888 &lt;parse&gt; time_format %iso8601 time_key logtime keep_time_key true &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you post following requests:</p> <pre><code># ruby code example require &#39;net/http&#39; require &#39;json&#39; record = {&#39;json&#39; =&gt; {&#39;k&#39; =&gt; &#39;hello1&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:27+09:00&#39;}.to_json} records = {&#39;json&#39; =&gt; [ {&#39;k&#39; =&gt; &#39;hello2&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:28+09:00&#39;}, {&#39;k&#39; =&gt; &#39;hello3&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:29+09:00&#39;} ].to_json} def post(path, params) http = Net::HTTP.new(&#39;127.0.0.1&#39;, 8888) req = Net::HTTP::Post.new(path, {}) req.set_form_data(params) http.request(req) end post(&quot;/test.http&quot;, record) post(&quot;/test.http&quot;, records) </code></pre> <p>the log shows expected result:</p> <pre><code>2020-06-10 11:14:27.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello1&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:27+09:00&quot;} 2020-06-10 11:14:28.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello2&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:28+09:00&quot;} 2020-06-10 11:14:29.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello3&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:29+09:00&quot;} </code></pre> <p>Before, this configuration doesn&#39;t work because <code>in_http</code> assumes the time value is floating point.</p> <h3>Major bug fixes</h3> <ul> <li>in_tail: Use actual path instead of base pattern for ignore list</li> <li>child_process helper: Fix child process failure due to SIGPIPE if the command uses stdout</li> </ul> <p>Enjoy logging!</p> 2020-06-22 <p>Hi users!</p> <p>We have released v1.11.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>in_http: Improve time field handling</h3> <p><code>in_http</code> now supports time parsing in record field for default json/msgpack request. Here is configuration example:</p> <pre><code>&lt;source&gt; @type http @id input_http port 8888 &lt;parse&gt; time_format %iso8601 time_key logtime keep_time_key true &lt;/parse&gt; &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you post following requests:</p> <pre><code># ruby code example require &#39;net/http&#39; require &#39;json&#39; record = {&#39;json&#39; =&gt; {&#39;k&#39; =&gt; &#39;hello1&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:27+09:00&#39;}.to_json} records = {&#39;json&#39; =&gt; [ {&#39;k&#39; =&gt; &#39;hello2&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:28+09:00&#39;}, {&#39;k&#39; =&gt; &#39;hello3&#39;, &#39;logtime&#39; =&gt; &#39;2020-06-10T11:14:29+09:00&#39;} ].to_json} def post(path, params) http = Net::HTTP.new(&#39;127.0.0.1&#39;, 8888) req = Net::HTTP::Post.new(path, {}) req.set_form_data(params) http.request(req) end post(&quot;/test.http&quot;, record) post(&quot;/test.http&quot;, records) </code></pre> <p>the log shows expected result:</p> <pre><code>2020-06-10 11:14:27.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello1&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:27+09:00&quot;} 2020-06-10 11:14:28.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello2&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:28+09:00&quot;} 2020-06-10 11:14:29.000000000 +0900 test.http: { &quot;k&quot;:&quot;hello3&quot;,&quot;logtime&quot;:&quot;2020-06-10T11:14:29+09:00&quot;} </code></pre> <p>Before, this configuration doesn&#39;t work because <code>in_http</code> assumes the time value is floating point.</p> <h3>Major bug fixes</h3> <ul> <li>in_tail: Use actual path instead of base pattern for ignore list</li> <li>child_process helper: Fix child process failure due to SIGPIPE if the command uses stdout</li> </ul> <p>Enjoy logging!</p> Fluentd v1.11.0 has been released https://www.fluentd.org/blog/fluentd-v1.11.0-has-been-released <p>Hi users!</p> <p>We have released v1.11.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>in_unix: Use v1 API</h3> <p>Migrate to v1 plugin API.</p> <p>In addition, <code>in_unix</code> now supports <code>tag</code> parameter to use fixed tag.</p> <h3>Improve syslog parser</h3> <p>There are several improvements:</p> <ul> <li>Improve <code>message_format auto</code> performance by avoiding object allocation</li> <li>Support any <code>time_format</code> for RFC3164 with <code>parser_type string</code> </li> <li>Support <code>parser_type string</code> for RFC5424. This parser is 2x faster than <code>parser_type regexp</code></li> </ul> <p>We now recommend to use <code>parser_type string</code> for both RFC3164 and RFC5424.</p> <h3>Major bug fixes</h3> <ul> <li>in_gc_stat: Add <code>use_symbol_keys</code> parameter to support non-stdout plugin.</li> </ul> <p>Enjoy logging!</p> 2020-06-04 <p>Hi users!</p> <p>We have released v1.11.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>.</p> <h3>in_unix: Use v1 API</h3> <p>Migrate to v1 plugin API.</p> <p>In addition, <code>in_unix</code> now supports <code>tag</code> parameter to use fixed tag.</p> <h3>Improve syslog parser</h3> <p>There are several improvements:</p> <ul> <li>Improve <code>message_format auto</code> performance by avoiding object allocation</li> <li>Support any <code>time_format</code> for RFC3164 with <code>parser_type string</code> </li> <li>Support <code>parser_type string</code> for RFC5424. This parser is 2x faster than <code>parser_type regexp</code></li> </ul> <p>We now recommend to use <code>parser_type string</code> for both RFC3164 and RFC5424.</p> <h3>Major bug fixes</h3> <ul> <li>in_gc_stat: Add <code>use_symbol_keys</code> parameter to support non-stdout plugin.</li> </ul> <p>Enjoy logging!</p> Fluentd v1.10.4 has been released https://www.fluentd.org/blog/fluentd-v1.10.4-has-been-released <p>Hi users!</p> <p>We have released v1.10.4. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release fixes several bugs.</p> <h3>out_http: Support single json array payload</h3> <p><code>out_http</code> uses <code>ndjson</code> for json payload by default.</p> <pre><code>{&quot;key&quot;:&quot;value1&quot;} {&quot;key&quot;:&quot;value2&quot;} ... {&quot;key&quot;:&quot;valueN&quot;} </code></pre> <p>You can change payload format to single json array by setting <code>json_array true</code>.</p> <pre><code># conf &lt;match pattern&gt; @type http # other parameters... json_array true &lt;format&gt; @type json add_newline false &lt;/format&gt; &lt;/match&gt; # payload [{&quot;key&quot;:&quot;value1&quot;},{&quot;key&quot;:&quot;value2&quot;},...,{&quot;key&quot;:&quot;valueN&quot;}] </code></pre> <h3>Major bug fixes</h3> <ul> <li>supervisor: Call <code>File.umask(0)</code> for standalone worker</li> <li>out_forward: Fix <code>ZeroDivisionError</code> issue with <code>weight 0</code></li> </ul> <p>Enjoy logging!</p> 2020-05-12 <p>Hi users!</p> <p>We have released v1.10.4. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release fixes several bugs.</p> <h3>out_http: Support single json array payload</h3> <p><code>out_http</code> uses <code>ndjson</code> for json payload by default.</p> <pre><code>{&quot;key&quot;:&quot;value1&quot;} {&quot;key&quot;:&quot;value2&quot;} ... {&quot;key&quot;:&quot;valueN&quot;} </code></pre> <p>You can change payload format to single json array by setting <code>json_array true</code>.</p> <pre><code># conf &lt;match pattern&gt; @type http # other parameters... json_array true &lt;format&gt; @type json add_newline false &lt;/format&gt; &lt;/match&gt; # payload [{&quot;key&quot;:&quot;value1&quot;},{&quot;key&quot;:&quot;value2&quot;},...,{&quot;key&quot;:&quot;valueN&quot;}] </code></pre> <h3>Major bug fixes</h3> <ul> <li>supervisor: Call <code>File.umask(0)</code> for standalone worker</li> <li>out_forward: Fix <code>ZeroDivisionError</code> issue with <code>weight 0</code></li> </ul> <p>Enjoy logging!</p> Fluentd v1.10.3 has been released https://www.fluentd.org/blog/fluentd-v1.10.3-has-been-released <p>Hi users!</p> <p>We have released v1.10.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release fixes several bugs.</p> <h3>record accessor helper: Support <code>set</code> method</h3> <p>This feature is for record mutation in the plugin. Here is code example:</p> <pre><code>record = {&#39;key1&#39; =&gt; {&#39;key2&#39; =&gt; &quot;value&quot;}} accessor = record_accessor_create(&quot;$[&#39;key1&#39;][&#39;key2&#39;]&quot;) accessor.set(r, &quot;new value!&quot;) # record is {&#39;key1&#39; =&gt; {&#39;key2&#39; =&gt; &quot;new value!&quot;}} </code></pre> <h3>config: Ruby DSL format is now deprecated</h3> <p>We decided to deprecate experimental Ruby DSL configuration by following reasons:</p> <ul> <li>Hard to maintain full configuation features</li> <li>Hard to implement/parse in other system</li> </ul> <h3>Major bug fixes</h3> <ul> <li>out<em>forward: Disable `linger</em>timeout` setting on Windows</li> <li>service discovery: Fix warning of service discovery manager when fluentd stops</li> </ul> <p>Enjoy logging!</p> 2020-05-06 <p>Hi users!</p> <p>We have released v1.10.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release fixes several bugs.</p> <h3>record accessor helper: Support <code>set</code> method</h3> <p>This feature is for record mutation in the plugin. Here is code example:</p> <pre><code>record = {&#39;key1&#39; =&gt; {&#39;key2&#39; =&gt; &quot;value&quot;}} accessor = record_accessor_create(&quot;$[&#39;key1&#39;][&#39;key2&#39;]&quot;) accessor.set(r, &quot;new value!&quot;) # record is {&#39;key1&#39; =&gt; {&#39;key2&#39; =&gt; &quot;new value!&quot;}} </code></pre> <h3>config: Ruby DSL format is now deprecated</h3> <p>We decided to deprecate experimental Ruby DSL configuration by following reasons:</p> <ul> <li>Hard to maintain full configuation features</li> <li>Hard to implement/parse in other system</li> </ul> <h3>Major bug fixes</h3> <ul> <li>out<em>forward: Disable `linger</em>timeout` setting on Windows</li> <li>service discovery: Fix warning of service discovery manager when fluentd stops</li> </ul> <p>Enjoy logging!</p> Fluentd v1.10.2 has been released https://www.fluentd.org/blog/fluentd-v1.10.2-has-been-released <p>Hi users!</p> <p>We have released v1.10.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release fixes several bugs.</p> <h3>socket: Allow certificate chains in client certificate</h3> <p><code>socket</code> helper based plugin now accepts certificate chains for client certificate. For example, out_forward&#39;s <code>tls_client_cert_path</code> now accepts certificate chains.</p> <h3>log: Add <code>ignore_repeated_log_interval</code> parameter</h3> <p>Specify interval to ignore repeated log/stacktrace messages like below.</p> <pre><code>&lt;system&gt; ignore_repeated_log_interval 2s &lt;/system&gt; </code></pre> <p>Under high-load environment, output destination sometimes becomes unstable and it generates lots of logs with same message. This parameter mitigates such situation.</p> <h3>Major bug fixes</h3> <ul> <li>out_forward: Fix obsolete socket leak when <code>keepalive_timeout</code> is not set</li> <li>command: Fix regression of supervisor&#39;s worker and <code>--daemon</code> option combo</li> <li>command: Fix <code>fluent-debug</code> start failure</li> </ul> <p>Enjoy logging!</p> 2020-04-15 <p>Hi users!</p> <p>We have released v1.10.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release fixes several bugs.</p> <h3>socket: Allow certificate chains in client certificate</h3> <p><code>socket</code> helper based plugin now accepts certificate chains for client certificate. For example, out_forward&#39;s <code>tls_client_cert_path</code> now accepts certificate chains.</p> <h3>log: Add <code>ignore_repeated_log_interval</code> parameter</h3> <p>Specify interval to ignore repeated log/stacktrace messages like below.</p> <pre><code>&lt;system&gt; ignore_repeated_log_interval 2s &lt;/system&gt; </code></pre> <p>Under high-load environment, output destination sometimes becomes unstable and it generates lots of logs with same message. This parameter mitigates such situation.</p> <h3>Major bug fixes</h3> <ul> <li>out_forward: Fix obsolete socket leak when <code>keepalive_timeout</code> is not set</li> <li>command: Fix regression of supervisor&#39;s worker and <code>--daemon</code> option combo</li> <li>command: Fix <code>fluent-debug</code> start failure</li> </ul> <p>Enjoy logging!</p>