Fluentd joins the Cloud Native Computing Foundation

Fluentd is giving a big step forward in terms of adoption. It was not a surprise that it was early adopted by several companies and from a cloud perspective, it become the default standard to solve Logging in containerized environments. We were able to see how Docker users relies on Fluentd to scale logging while in the orchestration area, users from Kubernetes started their own integrations too, nowadays Kubernetes uses Fluentd to ship logs to Elasticsearch and Google Cloud Platform. Fluentd is having an organic grow.

From a project perspective, adoption is the key to succeeed. But to accomplish this adoption a project needs to be a good citizen with other components, a good integration is always desired. Fluentd adoption is merely thanks to the flexibility to adapt and integrate with other platforms, and of course solving a real-worl problem which is logging.

As you might noticed, in the last two years Fluentd team was very active sharing logging knowledge in several conference around the world, I'd say our biggest participation have been in LinuxCon Asia, Europe and NorthAmerica (in all it versions!). This kind of interaction was really positive to understand how the project could evolve even more. At the beginning of 2016, we were thinking about what would be the next natural step for Fluentd and we saw some lights when Linux Foundation announced the new Cloud Native Computing Foundation (aka CNCF).

Since CNCF is a nonprofit organization committed to advancing the development of cloud native technology, it sound a really good fit for Fluentd, at that moment Google already donated Kubernetes to CNCF. One of the biggest benefits of CNCF compared to other foundations are:

  • Flexibility: development and roadmap continue being handled by the project. CNCF do not want to control that, so we can move forward very quickly.
  • Ecosystem: formally Fluentd can be part of a cloud native stack. Work together with Kubernetes team (and others) in a better way.
  • Investment: CNCF donates resources for documentation and give access to the CNCF Cluster (1k servers).
  • Awareness: Fluentd becomes recognized as the default Logging Cloud Native technology.

After a long process of review and technical discussions, the CNCF Technical Oversight Committee voted for Fluentd ending in positive result, Fluentd joins the CNCF and this was announced in the Opening Keynote session at CloudNativeCon (jump to minute 24:40):

Fluentd at CNCF

Note that Fluentd is a whole ecosystem, if you look around inside our Github Organization, you will see around 35 repositories including Fluentd service, plugins, languages SDKs and complement project such as Fluent Bit. All of them are part of CNCF now!.

In name of Treasure Data, I want thanks to every developer of Fluentd, plugins and SDKs who were very supportive on this transition for the project. Our code and community is part of something bigger now :) .

What's Next ?

Fluentd team continue working hard to make it even better, there is a long roadmap for v0.14 and looking forward for a v1.0 on Q1 of 2017. We want everybody continue be involved on this, this is a really exciting time in the Cloud Native Era and Fluentd Community is having a key role on it.

Read More

Fluentd v0.14.9 has been released

Hi users!

We have just shipped Fluentd v0.14.9 including built-in plugin migration and bug fixes.

Here are major changes (full ChangeLog is available here):

We say again, fluentd v0.14 is still development version. If you try to use v0.14, check your configuration and plugins carefully.

Migrate several plugins to v0.14 API

We continue to migrate built-in plugins to v0.14 API. Here are migrated plugins in this release:

  • in_http
  • in_forward
  • out_forward
  • out_file
  • out_exec
  • out_exec_filter

We describe important changes of these plugins.

in_http

We removed un-documented detach process feature from in_http because DetachMultiProcessMixin module is deprecated in v0.14. If you set detach process related parameters in your configuration, it is ignored.

out_forward

Since this version, time_as_integer parameter is now false. It means v0.14.9's out_forward can't forward data to v0.12's in_forward by default. You need to set time_as_integer true explicitly. We have already mentioned this point in v0.14.0 release, so we hope this change doesn't break your v0.14 to v0.12 forwarding.

out_file

v0.14 Plugin API provides placeholder feature. You can emulate fluent-plugin-forest by more flexible way. Here is configuration example:

<match mydata>
  @type file
  path /path/to/${key}
  <buffer time,key>
    flush_at_shutdown true
  </buffer>
  <format>
    @type json
  </format>
</match>

${key} refers the key field of event record. If you pass {"key":"foo"} record, actual output file becomes /path/to/foo. The important point is if you want to refer time, tag or record keys on path, you need to list keys in <buffer CHUNK_KEYS> (See also v0.14 Plugin API slide). Here are several examples:

# Popular time, tag and key case
<match mydata>
  @type file
  path /path/to/%Y/%m/%d/${tag}/${key} # path is /path/to/2016/11/15/mydata/foo_0.log
  <buffer time,tag,key>
    flush_at_shutdown true
  </buffer>
  # ...
</match>

# Of course, you can use more keys
<match mydata>
  @type file
  path /path/to/${key1}/${key2}/{key3} # If record is {"key1":"foo","key2":"bar","key3":"baz"}, path is /path/to/foo/bar/baz.20161115_0.log
  <buffer time,key1,key2,key3>
    flush_at_shutdown true
  </buffer>
  # ...
</match>

out_file plugin requires time in CHUNK_KEYS because its placeholder is used in path implicitly.

We have a plan to migrate other 3rd party plugins to v0.14, e.g s3, kafka and more. We can say "Goodbye fluent-plugin-forest!" in near future.

Port Parser filter plugin

fluent-plugin-parser plugin is widely used in the world, so we decided to port this plugin into the core.

Note that we changed invalid event handling. fluent-plugin-parser logs warning when log is invalid. On the other hand, built-in parser filter emits invalid events to built-in @ERROR label. In this result, you can process invalid events using other plugins.

<source>
  @type forward
</source>

<filter app.**>
  @type parser
  key_name log
  <parse>
    @type json
  </parse>
</filter>

# If log field is json, record comes here
<match app.**>
  @type stdout
</match>

<label @ERROR>
  # If log field is not json, record comes here. Store such events into local file.
  <match app.**>
    @type file
    # ...
  </match>
</label>

record_transformer filter: Change default behaviours

record_transformer changes and removes old behaviours.

  • auto_typecast is now true. It means the result of ${10 - 2} or ${record["int_field"]} is integer, not string, by default.
  • Remove ${tags} placeholder. Use tag_parts instead.

We also have a plan to remove ${key} placeholder in the next version. Use ${record["key"]} instead.

Major bug fixes

  • fluent-cat: Fix fluent-cat command to send sub-second precision time #1277
  • out_forward: fix not to raise error when out_forward is initialized as secondary #1313

Enjoy logging!

Read More

Fluentd v0.14.7 has been released

Hi users!

We have just shipped Fluentd v0.14.7. It's the release after a month from last release, including some major feature updates and bug fixes.

Here are major changes (full ChangeLog is available here):

Data compression support in buffers and forward plugin

Now buffer plugins support to write events into chunk as compressed data! It can save memory/disk spaces (especially under retrying situation), using additional CPU time. This feature is available with all buffer plugin types (both of memory and file).

<match mydata.*>
  @type any_output_plugin
  <buffer>
    @type memory
    compress gzip
    chunk_limit_size 16m  # 16MB compressed events
    chunk_total_size 512m # 512MB compressed events in total
  </buffer>
</match>

Fluentd compresses events before writing these into buffer chunk, and extract these data before passing these to output plugins. Output plugins can handle these data without any modification. After some very rough evaluation, we expect that compressed buffer consumes about 30% of memory/disk space against non-compressed buffers. It's larger than we expects for normal gzip, because Fluentd's buffer compression are done on relatively small amount of data units.

Moreover, forward input/output plugins also supports compression, to trasfer data in compressed format.

# In edge/forwarder node
<match mydata.*>
  @type forward
  compress gzip
  <server>
    host myserver.local # this Fluentd must be v0.14.7 or later!
    port 24224
  </server>
  # buffer compression is automatically turned on.
</match>

# In aggregator node
<source>
  @type forward
  port 24224
</source>

Forward output plugin automatically turn buffer compression on, and transfer the raw (compressed) bytes to destination (forward input plugin) with compressed option in forward protocol.

The common and symmetric time parser/formatter parameters

There were many fragmented configuration parameters to control time format and timezone. At v0.14.7, these parameters were unified and just a set of parameters are provided in formatter, parser and input/output/filter plugins which supports inject/extract plugin helpers.

Available configuration parameters are:

  • time_type: type of time representation [string, unixtime, float]
  • time_format: time format of string representation (like %Y-%m-%d %H:%M:%S.%N %z)
  • localtime: boolean value to specify to use local time zone
  • utc: boolean value to specify to use UTC
  • timezone: string to specify timezone, in any format of [+-]HH:MM, [+-]HHMM, [+-]HH or Region/Zone (e.g., Asia/Tokyo)

These parameters are available in all parser/formatter plugins (if that plugin supports time parsing/formatting).

Log rotation support about Fluentd's log file

Two new command line options were introduced to make it possible to rotate logs of Fluentd itself!

-o, --log PATH                   log file path
    --log-rotate-age AGE         generations to keep rotated log files
    --log-rotate-size BYTES      sets the byte size to rotate log files

There is -o(--log) option for a long time to write Fluentd's log into a file, but it can't be rotated automatically (There were just one way to do it - use SIGUSR1). On Fluentd v0.14.7, you can use --log-rotate-age and/or --log-rotate-size to rotate log files per specified size, and leave old log files within specified ages. Log rotation is enabled when at least one of these parameters are specified: --log-rotate-age(5 if not specified), --log-rotate-size(1MB if not specified).

Secondary plugin to dump backup data, and a tool to load it

When buffered output plugin continues to fail writing data to destination, we can use <secondary> section to dump that data as backup... but there were no best practice or good way to open/read/modify these dumped files.

Fluentd v0.14.7 has a new output plugin, out_secondary_file, which is only for secondary output. That plugin has just few configuration parameters, to dump buffered data to a specified directory.

<match failing.data>
  @type forward
  <buffer tag>
    flush_interval 1s
  </buffer>
  <server>
    host unreachable.destination.local
  </server>
  <secondary>
    @type secondary_file
    directory /my/backup/${tag}
    # basename dump.bin # expected full path is /my/backup/failing.data/dump.bin.0
    # append   false
    # compress text     # or gzip
  </secondary>
</match>

When forward plugin fails to send data in enough times, Fluentd uses secondary_file plugin to write buffered data into /my/backup/failing.data directory.

The format of dumped data depends on the format method of primary output plugin. If the primary output plugin formats data into JSON, you can get a JSON object per lines. But many plugin uses Fluentd's standard format, MessagePack, as a format of buffer chunk data. It's just binary data, not human readable. Now we have a tool new introduced, fluent-binlog-reader, which is to read such binary data from files, and format to reuse these data in our own scripts or tools.

$ fluent-binlog-reader head dump.bin.0
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}
2016-08-12T17:24:18+09:00       dump.bin.0      {"message":"dummy"}

The default format is just same with stdout output plugin, and all formats supported in Fluentd are available via -f option.

Major bug fixes

  • in_tail: fix a bug to crash to read large amount logs #1229
  • out_forward: fix a bug that UDP heartbeat doesn't work #1238
  • Fix a bug not to stop enqueue/flush threads correctly #1264
  • Fix a bug to call shutdown method (and some others) twice #1161, #1222

Enjoy logging!

Read More

Fluentd v0.14.6 has been released

Hi users!

We have released Fluentd version 0.14.6.

This is emergency release for fixing regressions, so not including new features.

Bug fixes

  • in_tail: Add a missing parser_multiline require: #1212
  • forward: Mark secret parameters of forward plugins as secret: #1209

Enjoy logging!

Read More

Fluentd v0.14.5 has been released

Hi users!

6 days after the last relase, we shipped the latest release, Fluentd v0.14.5. It's a bit quick release from the last one, to fix a problem which affects many people who tries Fluentd with existing configurations.

This release also includes some feature improvements. The biggest one is the update of Fluentd forward protocol and forward input/output plugins, to support authentication and authorization.

Here are major changes (full ChangeLog is available here):

forward plugins: Update protocol to v1

Fluentd forward plugins are to transfer Fluentd's events from one node to another via network. It's widely used by Fluentd processes, Docker logging driver for Fluentd, Fluent logger for programming languages and many others.

Before v0.14.5, Forward plugins/protocol was very simple one, to transfer events, without providing any safety, including network ACL, shared keys or password authentication. It was called as Forward Protocol version 0. Anyone who can connect TCP ports of in_forward can inject events into Fluentd processes on that version.

At v0.14.5, forward input/output plugins support Fluentd Forward Protocol version 1. This new protocol is perfectly compatible with older version 0, and provides the features below in addition to it.

  • Server (in_forward) authentication using shared key
  • Client (out_forward) authorization using shared key (per client) and pair of username/password

Once in_forward is configured with <security> section, the only nodes which are configured with correct shared_key and username/password can connect to that node.

# in_forward
<source>
  @type forward
  port 24224
  <secure>
    self_hostname aggregator1.fluentd.local
    shared_key    my_secret_string
    user_auth     true
    <user>
      username user1
      password pass1
    </user>
    <user>
      username user2
      password pass2
    </user>
    <client>
      network 10.0.0.0/16
      users user1,user2
    </client>
    <client>
      # more secure area
      network 192.168.0.0/24
      shared_key more_secret_string
      useers user1
    </client>
  </secure>
</source>

# out_forward
<match secure.data.*>
  @type forward
  <security>
    self_hostname edge110.fluentd.local
    shared_key    my_secret_string
  </security>
  <server>
    host 10.0.0.1
    port 24224
    username user1
    password pass1
  </server>
</match>

For now, This version doesn't provide encryption like SSL/TLS. So this authentication/authorization will be done on raw TCP transportation. Forward plugins never send raw shared keys and passwords... but you may have to wait next (or more) released to support SSL/TLS communication if you want more secure configurations (it's in our plan).

One good news: Fluentd forward protocol v1 is just same with the protocol implemented on fluent-plugin-secure-forward (except for TCP or SSL/TLS). So you can configure the total architecture like below using out_secure_forward, in_forward(v0.14.5) and SSL/TLS terminator like AWS ELB.

[out_secure_forward] ---> (SSL/TLS over internet) ---> [AWS ELB] ---> (TCP in AWS VPC) ---> [in_forward]

This configuration works pretty well, because Fluentd doesn't consume its CPU for encryption and decryption.

Major bug fixes

  • fix to raise configuration error explicitly for missing @type in configuration file #1202
  • fix bug to fail to launch Fluentd when configuration uses v0.12 MultiOutput plugins #1206

Enjoy logging!

Read More

About Fluentd

Fluentd is an open source data collector to simplify log management.

Learn

Want to learn more about Fluentd? Check out these pages.

Follow Us!