Fluentd Blog https://www.fluentd.org/blog Get the Latest Updates on Fluentd and More Fluentd has Graduated! https://www.fluentd.org/blog/fluentd-cncf-graduation <p>We are thrilled to announce that <a href="https://www.fluentd.org">Fluentd</a> has become the sixth <a href="https://www.cncf.io/announcement/2019/04/11/cncf-announces-fluentd-graduation">CNCF Graduated Project</a>! This is a huge recognition to the project and its community. Fluentd wide adoption by the Enterprise and contributions are in a continuous grow: we have more than 900 contributed plugins and more than 100k downloads a day!</p> <p><em>Fluentd was created in 2011 by <a href="http://github.com/frsyuki">Sadayuki “Sada” Furuhashi</a>, co-founder of <a href="https://www.treasuredata.com/">Treasure Data, Inc.</a>, as an open source data collector for building a <a href="https://www.fluentd.org/blog/unified-logging-layer">Unified Logging Layer</a>, which unifies the data collection and consumption for better use and understanding of data. In November 2016, Fluentd was accepted as CNCF’s sixth hosted project after Kubernetes, Prometheus, and OpenTracing.</em></p> <p><a href="https://www.cncf.io/announcement/2019/04/11/cncf-announces-fluentd-graduation/"> <img src="/images/blog/fluentd_graduation.jpg" width="50%" height="50%"/> </a></p> <p>As of today, <a href="https://fluentd.org">Fluentd</a> is used by major companies to solve log collection and processing at scale: Microsoft, Redhat, and Google are a few names of its known users.</p> <p>Fluentd&#39;s core maintainer, <a href="https://github.com/repeatedly">Masahiro Nakagawa</a> (well known as <a href="https://twitter.com/repeatedly">@repeatedly</a> on Twitter), said that he&#39;s thrilled to have had the opportunity to collaborate with the CNCF community and to grow its user base, and it&#39;s an honor to see Fluentd graduated. Today we celebrate this big accomplishment, and we continue looking forward to improving on upcoming Logging and data processing challenges.</p> <p><strong>Fluentd at Conferences on 2019</strong></p> <p>Fluentd team will be attending some conferences this year, save the date!</p> <ul> <li><a href="https://events.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019">KubeCon Europe, Barcelona 2019. May 20 - 23</a></li> <li><a href="https://www.lfasiallc.com/events/kubecon-cloudnativecon-china-2019">KubeCon China, Shanghai 2019, June 25 - 26</a></li> <li>Fluentd Meetup!, Tokyo Japan 2019, July (date to be defined)</li> <li><a href="https://events.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2019">KubeCon NorthAmerica, San Diego California - US, November 18 - 21</a></li> </ul> 2019-04-11 <p>We are thrilled to announce that <a href="https://www.fluentd.org">Fluentd</a> has become the sixth <a href="https://www.cncf.io/announcement/2019/04/11/cncf-announces-fluentd-graduation">CNCF Graduated Project</a>! This is a huge recognition to the project and its community. Fluentd wide adoption by the Enterprise and contributions are in a continuous grow: we have more than 900 contributed plugins and more than 100k downloads a day!</p> <p><em>Fluentd was created in 2011 by <a href="http://github.com/frsyuki">Sadayuki “Sada” Furuhashi</a>, co-founder of <a href="https://www.treasuredata.com/">Treasure Data, Inc.</a>, as an open source data collector for building a <a href="https://www.fluentd.org/blog/unified-logging-layer">Unified Logging Layer</a>, which unifies the data collection and consumption for better use and understanding of data. In November 2016, Fluentd was accepted as CNCF’s sixth hosted project after Kubernetes, Prometheus, and OpenTracing.</em></p> <p><a href="https://www.cncf.io/announcement/2019/04/11/cncf-announces-fluentd-graduation/"> <img src="/images/blog/fluentd_graduation.jpg" width="50%" height="50%"/> </a></p> <p>As of today, <a href="https://fluentd.org">Fluentd</a> is used by major companies to solve log collection and processing at scale: Microsoft, Redhat, and Google are a few names of its known users.</p> <p>Fluentd&#39;s core maintainer, <a href="https://github.com/repeatedly">Masahiro Nakagawa</a> (well known as <a href="https://twitter.com/repeatedly">@repeatedly</a> on Twitter), said that he&#39;s thrilled to have had the opportunity to collaborate with the CNCF community and to grow its user base, and it&#39;s an honor to see Fluentd graduated. Today we celebrate this big accomplishment, and we continue looking forward to improving on upcoming Logging and data processing challenges.</p> <p><strong>Fluentd at Conferences on 2019</strong></p> <p>Fluentd team will be attending some conferences this year, save the date!</p> <ul> <li><a href="https://events.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019">KubeCon Europe, Barcelona 2019. May 20 - 23</a></li> <li><a href="https://www.lfasiallc.com/events/kubecon-cloudnativecon-china-2019">KubeCon China, Shanghai 2019, June 25 - 26</a></li> <li>Fluentd Meetup!, Tokyo Japan 2019, July (date to be defined)</li> <li><a href="https://events.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2019">KubeCon NorthAmerica, San Diego California - US, November 18 - 21</a></li> </ul> Fluentd v1.4.2 has been released https://www.fluentd.org/blog/fluentd-v1.4.2-has-been-released <p>Hi users!</p> <p>We have released v1.4.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements and bug fixes.</p> <h3>in_monitor_agent: Expose current timekey list as a buffer metrics</h3> <p><code>in_monitor_agent</code> now returns new <code>buffer_timekeys</code> field like below:</p> <pre><code>{ &quot;buffer_queue_length&quot;:1, &quot;buffer_timekeys&quot; = &gt;[1554741360], ... } </code></pre> <p>You can use this field to check current chunk status, e.g. delay, stuck or more.</p> <h3>in_tcp/in_udp: Add <code>source_address_key</code> parameter</h3> <p>This is same as syslog input plugin.</p> <pre><code>&lt;source&gt; @type tcp source_address_key addr # other parameters... &lt;/source&gt; </code></pre> <p>With this configuation, client&#39;s address is assigned to <code>addr</code> field.</p> <h3>Major bug fixes</h3> <ul> <li>in_tail: Create parent directories for symlink</li> <li>log: Fix time format handling of plugin logger when log format is JSON</li> </ul> <p>Enjoy logging!</p> 2019-04-02 <p>Hi users!</p> <p>We have released v1.4.2. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements and bug fixes.</p> <h3>in_monitor_agent: Expose current timekey list as a buffer metrics</h3> <p><code>in_monitor_agent</code> now returns new <code>buffer_timekeys</code> field like below:</p> <pre><code>{ &quot;buffer_queue_length&quot;:1, &quot;buffer_timekeys&quot; = &gt;[1554741360], ... } </code></pre> <p>You can use this field to check current chunk status, e.g. delay, stuck or more.</p> <h3>in_tcp/in_udp: Add <code>source_address_key</code> parameter</h3> <p>This is same as syslog input plugin.</p> <pre><code>&lt;source&gt; @type tcp source_address_key addr # other parameters... &lt;/source&gt; </code></pre> <p>With this configuation, client&#39;s address is assigned to <code>addr</code> field.</p> <h3>Major bug fixes</h3> <ul> <li>in_tail: Create parent directories for symlink</li> <li>log: Fix time format handling of plugin logger when log format is JSON</li> </ul> <p>Enjoy logging!</p> Fluentd v1.4.1 has been released https://www.fluentd.org/blog/fluentd-v1.4.1-has-been-released <p>Hi users!</p> <p>We have released v1.4.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements and bug fixes.</p> <h3>system: Add <code>worker_id</code> to <code>process_name</code> when workers is larger than 1</h3> <p>This improves the multi-workers operation.</p> <pre><code>&lt;system&gt; process_name fluentd_aggr workers 2 &lt;/system&gt; </code></pre> <p>With this configuration, older version shows same process name for workers.</p> <pre><code># ps result repeatedly 94982 0.0 0.2 2521484 29948 s004 S+ 9:56PM 0:00.48 worker:fluentd_aggr repeatedly 94983 0.0 0.2 2522508 28304 s004 S+ 9:56PM 0:00.49 worker:fluentd_aggr </code></pre> <p>Since v1.4.1, process name has worker id to identify the process.</p> <pre><code># ps result repeatedly 94879 0.0 0.2 2522508 28808 s004 S+ 9:55PM 0:00.49 worker:fluentd_aggr0 repeatedly 94880 0.0 0.2 2521484 29084 s004 S+ 9:55PM 0:00.49 worker:fluentd_aggr1 </code></pre> <h3>Major bug fixes</h3> <ul> <li>out_forward: Make <code>tls_client_private_key_passphrase</code> secret</li> <li>in_syslog: Check message length when read from buffer in octet counting</li> </ul> <p>Enjoy logging!</p> 2019-03-18 <p>Hi users!</p> <p>We have released v1.4.1. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes several enhancements and bug fixes.</p> <h3>system: Add <code>worker_id</code> to <code>process_name</code> when workers is larger than 1</h3> <p>This improves the multi-workers operation.</p> <pre><code>&lt;system&gt; process_name fluentd_aggr workers 2 &lt;/system&gt; </code></pre> <p>With this configuration, older version shows same process name for workers.</p> <pre><code># ps result repeatedly 94982 0.0 0.2 2521484 29948 s004 S+ 9:56PM 0:00.48 worker:fluentd_aggr repeatedly 94983 0.0 0.2 2522508 28304 s004 S+ 9:56PM 0:00.49 worker:fluentd_aggr </code></pre> <p>Since v1.4.1, process name has worker id to identify the process.</p> <pre><code># ps result repeatedly 94879 0.0 0.2 2522508 28808 s004 S+ 9:55PM 0:00.49 worker:fluentd_aggr0 repeatedly 94880 0.0 0.2 2521484 29084 s004 S+ 9:55PM 0:00.49 worker:fluentd_aggr1 </code></pre> <h3>Major bug fixes</h3> <ul> <li>out_forward: Make <code>tls_client_private_key_passphrase</code> secret</li> <li>in_syslog: Check message length when read from buffer in octet counting</li> </ul> <p>Enjoy logging!</p> Fluentd v1.4.0 has been released https://www.fluentd.org/blog/fluentd-v1.4.0-has-been-released <p>Hi users!</p> <p>We have released v1.4.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes nice features and improvements.</p> <h3>multiprocess: Support <code>&lt;worker N-M&gt;</code> syntax</h3> <p>This feature enables grouping workers. Here is configuration example:</p> <pre><code>&lt;system&gt; workers 5 &lt;/system&gt; &lt;worker 0-1&gt; &lt;source&gt; @type forward &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; &lt;/worker&gt; &lt;worker 2-4&gt; &lt;source&gt; @type tcp &lt;parse&gt; @type json &lt;/parse&gt; tag test &lt;/source&gt; &lt;match test&gt; @type stdout &lt;/match&gt; &lt;/worker&gt; </code></pre> <p>With this configuration, worker <code>0</code>/<code>1</code> launches <code>forward</code> input with <code>24224</code> port and worker <code>2</code>/<code>3</code>/<code>4</code> launches <code>tcp</code> input with <code>5170</code> port.</p> <h3>out_file: Support placeholders in <code>symlink_path</code> parameters</h3> <p>This change improves <code>symlink_path</code> usecase. You can tail multiple files based on placeholders. Here is a configuration and result example:</p> <pre><code># config &lt;source&gt; @type dummy tag dummy1 &lt;/source&gt; &lt;source&gt; @type dummy tag dummy2 &lt;/source&gt; &lt;match dummy*&gt; @type file path /tmp/logs/${tag} symlink_path /tmp/logs/current-${tag} &lt;buffer tag,time&gt; @type file &lt;/buffer&gt; &lt;/match&gt; # result $ tree /tmp/logs/ /tmp/logs/ ├── ${tag} │   ├── buffer.b57fb1dd96306dd0b308e094f7ec2228f.log │   ├── buffer.b57fb1dd96306dd0b308e094f7ec2228f.log.meta │   ├── buffer.b57fb1dd96339a870530991d4871cfe11.log │   └── buffer.b57fb1dd96339a870530991d4871cfe11.log.meta ├── current-dummy1 -&gt; /tmp/logs/${tag}/buffer.b57fb1dd96339a870530991d4871cfe11.log └── current-dummy2 -&gt; /tmp/logs/${tag}/buffer.b57fb1dd96306dd0b308e094f7ec2228f.log </code></pre> <h3>config: Support embedded ruby code in section argument.</h3> <p>You can use <code>&quot;#{ruby code}&quot;</code> syntax in <code>&lt;filter&gt;</code>/<code>&lt;match&gt;</code> argument. Let&#39;s see configuration example:</p> <pre><code>&lt;match &quot;app.#{ENV[&#39;FLUENTD_TAG&#39;]}&quot;&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you set <code>dev</code> to <code>FLUENTD_TAG</code> environment variable, this config becomes <code>&lt;match app.dev&gt;</code>.</p> <h3>Major bug fixes</h3> <ul> <li>server: Ignore EPIPE in TLS accept to avoid fluentd restart</li> <li>server: Ignore IOError and unrecoverable errors in UDP</li> </ul> <p>Enjoy logging!</p> 2019-02-25 <p>Hi users!</p> <p>We have released v1.4.0. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release includes nice features and improvements.</p> <h3>multiprocess: Support <code>&lt;worker N-M&gt;</code> syntax</h3> <p>This feature enables grouping workers. Here is configuration example:</p> <pre><code>&lt;system&gt; workers 5 &lt;/system&gt; &lt;worker 0-1&gt; &lt;source&gt; @type forward &lt;/source&gt; &lt;match test.**&gt; @type stdout &lt;/match&gt; &lt;/worker&gt; &lt;worker 2-4&gt; &lt;source&gt; @type tcp &lt;parse&gt; @type json &lt;/parse&gt; tag test &lt;/source&gt; &lt;match test&gt; @type stdout &lt;/match&gt; &lt;/worker&gt; </code></pre> <p>With this configuration, worker <code>0</code>/<code>1</code> launches <code>forward</code> input with <code>24224</code> port and worker <code>2</code>/<code>3</code>/<code>4</code> launches <code>tcp</code> input with <code>5170</code> port.</p> <h3>out_file: Support placeholders in <code>symlink_path</code> parameters</h3> <p>This change improves <code>symlink_path</code> usecase. You can tail multiple files based on placeholders. Here is a configuration and result example:</p> <pre><code># config &lt;source&gt; @type dummy tag dummy1 &lt;/source&gt; &lt;source&gt; @type dummy tag dummy2 &lt;/source&gt; &lt;match dummy*&gt; @type file path /tmp/logs/${tag} symlink_path /tmp/logs/current-${tag} &lt;buffer tag,time&gt; @type file &lt;/buffer&gt; &lt;/match&gt; # result $ tree /tmp/logs/ /tmp/logs/ ├── ${tag} │   ├── buffer.b57fb1dd96306dd0b308e094f7ec2228f.log │   ├── buffer.b57fb1dd96306dd0b308e094f7ec2228f.log.meta │   ├── buffer.b57fb1dd96339a870530991d4871cfe11.log │   └── buffer.b57fb1dd96339a870530991d4871cfe11.log.meta ├── current-dummy1 -&gt; /tmp/logs/${tag}/buffer.b57fb1dd96339a870530991d4871cfe11.log └── current-dummy2 -&gt; /tmp/logs/${tag}/buffer.b57fb1dd96306dd0b308e094f7ec2228f.log </code></pre> <h3>config: Support embedded ruby code in section argument.</h3> <p>You can use <code>&quot;#{ruby code}&quot;</code> syntax in <code>&lt;filter&gt;</code>/<code>&lt;match&gt;</code> argument. Let&#39;s see configuration example:</p> <pre><code>&lt;match &quot;app.#{ENV[&#39;FLUENTD_TAG&#39;]}&quot;&gt; @type stdout &lt;/match&gt; </code></pre> <p>If you set <code>dev</code> to <code>FLUENTD_TAG</code> environment variable, this config becomes <code>&lt;match app.dev&gt;</code>.</p> <h3>Major bug fixes</h3> <ul> <li>server: Ignore EPIPE in TLS accept to avoid fluentd restart</li> <li>server: Ignore IOError and unrecoverable errors in UDP</li> </ul> <p>Enjoy logging!</p> Fluentd v1.3.3 has been released https://www.fluentd.org/blog/fluentd-v1.3.3-has-been-released <p>Hi users!</p> <p>We have released v1.3.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is mainly for bug fixes</p> <h3>parser_syslog: Support RFC5424 timestamp without subseconds</h3> <p>Now, <code>parser_syslog</code> can accept RFC5424 timestampe with/without subseconds.</p> <pre><code>&lt;16&gt;1 2017-02-06T13:14:15Z 192.168.0.1 fluentd - - - Hi, from Fluentd! &lt;16&gt;1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi, from Fluentd with subseconds! </code></pre> <p>No additional setting for these events.</p> <h3>Major bug fixes</h3> <ul> <li>server: Ignore <code>ECONNRESET</code> in TLS accept to avoid fluentd restart</li> <li>log: Fix plugin logger ignores fluentd log event setting This fixes <code>@id</code> having plugins ignore <code>&lt;label @FLUENT_LOG&gt;</code> configuration.</li> </ul> <p>Enjoy logging!</p> 2019-01-07 <p>Hi users!</p> <p>We have released v1.3.3. ChangeLog is <a href="https://github.com/fluent/fluentd/blob/master/CHANGELOG.md">here</a>. This release is mainly for bug fixes</p> <h3>parser_syslog: Support RFC5424 timestamp without subseconds</h3> <p>Now, <code>parser_syslog</code> can accept RFC5424 timestampe with/without subseconds.</p> <pre><code>&lt;16&gt;1 2017-02-06T13:14:15Z 192.168.0.1 fluentd - - - Hi, from Fluentd! &lt;16&gt;1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi, from Fluentd with subseconds! </code></pre> <p>No additional setting for these events.</p> <h3>Major bug fixes</h3> <ul> <li>server: Ignore <code>ECONNRESET</code> in TLS accept to avoid fluentd restart</li> <li>log: Fix plugin logger ignores fluentd log event setting This fixes <code>@id</code> having plugins ignore <code>&lt;label @FLUENT_LOG&gt;</code> configuration.</li> </ul> <p>Enjoy logging!</p>