Fluentd Blog

Fluentd v1.5.0 has been released

Hi users!

We have released v1.5.0. ChangeLog is here. This release includes nice features and improvements.

out_forward: Support keepalive feature

<source>
  @type tail
  # ... other parameters
</source>

<match pattern>
  @type forward
  # ... other parameters
  keepalive true
  keepalive_timeout 60s # optional. Default is no timeout
</match>

forward output plugin now supports keepalive/keepalive_timeout parameters. This feature is useful to reduce the connection cost per flush. In addition, this feature is also good for load balancer situation.

in_http/in_syslog: Support TLS transport

These plugins now support <transport tls> configuration. Here is syslog input / rsyslog config example for TLS:

# fluentd
<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  <transport tls>
    ca_path /etc/pki/ca.pem
    cert_path /etc/pki/cert.pem
    private_key_path /etc/pki/key.pem
    private_key_passphrase PASSPHRASE
  </transport>
  tag system
</source>

# rsyslog
$DefaultNetstreamDriverCAFile /etc/pki/ca.pem
$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
*.* @@127.0.0.1:5140

http input supports same parameters with <transport tls>.

in_forward: Add tag/add_tag_prefix parameters

You can change incoming event tag in forward input without additional plugins.

<source>
  @type forward
  port 24230
  tag system1.log # change tag to system1.log
</source>

<source>
  @type forward
  port 24240
  add_tag_prefix system2 # add prefix to incoming tag, e.g. system2.app.log
</source>

For event routing, we can use label feature but it can't modify event tag. These parameters are useful when you want to organize incoming tag in the aggregator.

Major bug fixes

• out_forward: Don't use SO_LINGER on SSL/TLS WinSock.
• server: Fix recursive lock issue in TLSServer

Enjoy logging!

Read More

Fluentd documentation is now hosted on gitbook

Today, we have updated docs.fluentd.org to use gitbook. This change has several merits:

• Better Look and Feel (same as fluent-bit document)
• Use popular Markdown for the content
• Reduce maintenance cost

fluentd-docs-gitbook is new repository and old fluentd-docs repository is now deprecated and archived.

If you find something wrong on docs.fluentd.org, use fluentd-docs-gitbook for issues/PRs.

Read More

Fluentd has Graduated!

We are thrilled to announce that Fluentd has become the sixth CNCF Graduated Project! This is a huge recognition to the project and its community. Fluentd wide adoption by the Enterprise and contributions are in a continuous grow: we have more than 900 contributed plugins and more than 100k downloads a day!

Fluentd was created in 2011 by Sadayuki “Sada” Furuhashi, co-founder of Treasure Data, Inc., as an open source data collector for building a Unified Logging Layer, which unifies the data collection and consumption for better use and understanding of data. In November 2016, Fluentd was accepted as CNCF’s sixth hosted project after Kubernetes, Prometheus, and OpenTracing.

As of today, Fluentd is used by major companies to solve log collection and processing at scale: Microsoft, Redhat, and Google are a few names of its known users.

Fluentd's core maintainer, Masahiro Nakagawa (well known as @repeatedly on Twitter), said that he's thrilled to have had the opportunity to collaborate with the CNCF community and to grow its user base, and it's an honor to see Fluentd graduated. Today we celebrate this big accomplishment, and we continue looking forward to improving on upcoming Logging and data processing challenges.

Fluentd at Conferences on 2019

Fluentd team will be attending some conferences this year, save the date!

• KubeCon Europe, Barcelona 2019. May 20 - 23
• KubeCon China, Shanghai 2019, June 25 - 26
• Fluentd Meetup!, Tokyo Japan 2019, July (date to be defined)
• KubeCon NorthAmerica, San Diego California - US, November 18 - 21

Read More

Fluentd v1.4.2 has been released

Hi users!

We have released v1.4.2. ChangeLog is here. This release includes several enhancements and bug fixes.

in_monitor_agent: Expose current timekey list as a buffer metrics

in_monitor_agent now returns new buffer_timekeys field like below:

{
  "buffer_queue_length":1,
  "buffer_timekeys" = >[1554741360],
  ...
}

You can use this field to check current chunk status, e.g. delay, stuck or more.

in_tcp/in_udp: Add source_address_key parameter

This is same as syslog input plugin.

<source>
  @type tcp
  source_address_key addr
  # other parameters...
</source>

With this configuation, client's address is assigned to addr field.

Major bug fixes

• in_tail: Create parent directories for symlink
• log: Fix time format handling of plugin logger when log format is JSON

Enjoy logging!

Read More

Fluentd v1.4.1 has been released

Hi users!

We have released v1.4.1. ChangeLog is here. This release includes several enhancements and bug fixes.

system: Add worker_id to process_name when workers is larger than 1

This improves the multi-workers operation.

<system>
  process_name fluentd_aggr
  workers 2
</system>

With this configuration, older version shows same process name for workers.

# ps result
repeatedly       94982   0.0  0.2  2521484  29948 s004  S+    9:56PM   0:00.48 worker:fluentd_aggr  
repeatedly       94983   0.0  0.2  2522508  28304 s004  S+    9:56PM   0:00.49 worker:fluentd_aggr

Since v1.4.1, process name has worker id to identify the process.

# ps result
repeatedly       94879   0.0  0.2  2522508  28808 s004  S+    9:55PM   0:00.49 worker:fluentd_aggr0
repeatedly       94880   0.0  0.2  2521484  29084 s004  S+    9:55PM   0:00.49 worker:fluentd_aggr1

Major bug fixes

• out_forward: Make tls_client_private_key_passphrase secret
• in_syslog: Check message length when read from buffer in octet counting

Enjoy logging!

Read More