Fluentd v0.14.6 has been released

Hi users!

We have released Fluentd version 0.14.6.

This is emergency release for fixing regressions, so not including new features.

Bug fixes

  • in_tail: Add a missing parser_multiline require: #1212
  • forward: Mark secret parameters of forward plugins as secret: #1209

Enjoy logging!

Read More

Fluentd v0.14.5 has been released

Hi users!

6 days after the last relase, we shipped the latest release, Fluentd v0.14.5. It's a bit quick release from the last one, to fix a problem which affects many people who tries Fluentd with existing configurations.

This release also includes some feature improvements. The biggest one is the update of Fluentd forward protocol and forward input/output plugins, to support authentication and authorization.

Here are major changes (full ChangeLog is available here):

forward plugins: Update protocol to v1

Fluentd forward plugins are to transfer Fluentd's events from one node to another via network. It's widely used by Fluentd processes, Docker logging driver for Fluentd, Fluent logger for programming languages and many others.

Before v0.14.5, Forward plugins/protocol was very simple one, to transfer events, without providing any safety, including network ACL, shared keys or password authentication. It was called as Forward Protocol version 0. Anyone who can connect TCP ports of in_forward can inject events into Fluentd processes on that version.

At v0.14.5, forward input/output plugins support Fluentd Forward Protocol version 1. This new protocol is perfectly compatible with older version 0, and provides the features below in addition to it.

  • Server (in_forward) authentication using shared key
  • Client (out_forward) authorization using shared key (per client) and pair of username/password

Once in_forward is configured with <security> section, the only nodes which are configured with correct shared_key and username/password can connect to that node.

# in_forward
<source>
  @type forward
  port 24224
  <secure>
    self_hostname aggregator1.fluentd.local
    shared_key    my_secret_string
    user_auth     true
    <user>
      username user1
      password pass1
    </user>
    <user>
      username user2
      password pass2
    </user>
    <client>
      network 10.0.0.0/16
      users user1,user2
    </client>
    <client>
      # more secure area
      network 192.168.0.0/24
      shared_key more_secret_string
      useers user1
    </client>
  </secure>
</source>

# out_forward
<match secure.data.*>
  @type forward
  <security>
    self_hostname edge110.fluentd.local
    shared_key    my_secret_string
  </security>
  <server>
    host 10.0.0.1
    port 24224
    username user1
    password pass1
  </server>
</match>

For now, This version doesn't provide encryption like SSL/TLS. So this authentication/authorization will be done on raw TCP transportation. Forward plugins never send raw shared keys and passwords... but you may have to wait next (or more) released to support SSL/TLS communication if you want more secure configurations (it's in our plan).

One good news: Fluentd forward protocol v1 is just same with the protocol implemented on fluent-plugin-secure-forward (except for TCP or SSL/TLS). So you can configure the total architecture like below using out_secure_forward, in_forward(v0.14.5) and SSL/TLS terminator like AWS ELB.

[out_secure_forward] ---> (SSL/TLS over internet) ---> [AWS ELB] ---> (TCP in AWS VPC) ---> [in_forward]

This configuration works pretty well, because Fluentd doesn't consume its CPU for encryption and decryption.

Major bug fixes

  • fix to raise configuration error explicitly for missing @type in configuration file #1202
  • fix bug to fail to launch Fluentd when configuration uses v0.12 MultiOutput plugins #1206

Enjoy logging!

Read More

Fluentd v0.12.29 has been released

Hi users!

We have released Fluentd version 0.12.29. Here are the changes:

New features / Enhancement

  • in_tail: Add from_encoding parameter: #1067
  • filter: Port parser filter to v0.12: #1203
  • config: New config_param options to specify deprecated/obsoleted parameters: #1186

Bug fixes

  • config: fix to raise ConfigError for type missing, and its tests: #1202
  • filter_record_transformer: Fix to prevent overwriting reserved placeholder keys such as tag: #1176

Port parser filter

parser filter of fluent-plugin-parser is for parsing one field of event record and updates event record itself with parsed result. This is useful for some use cases, e.g. docker logging driver, lazy log parsing at aggregator and etc.

Now, these use cases are popular and many users install fluent-plugin-parser on their environment. So we decided to port parser filter into core plugins to reduce extra setup cost.

Here is a configuration example for Docker loggin driver:

<filter docker.**>
  @type parser
  key_name log
  format json
  reserve_data true
</filter>

config: New config_param options to specify deprecated/obsoleted parameters

This is for plugin developers. Sometimes we want to deprecate plugin parameters, e.g. replace internal implementation with new library, add more useful parameter, etc. For such cases, we introduced new 2 options to config_param.

  • deprecated: fluentd shows warn message for deprecated parameter. A plugin should work with deprecated.
  • obsoleted: fluentd raises ConfigError for obsoleted parameter.

These options accept string value as a message. Here is code example:

config_param :param1, :integer, default: 10, deprecated: "User new_param1 instead"
config_param :param2, :string, default: nil, obsoleted: "Use new_param2 instead"

Here is log example:

2016-09-07 15:50:16 +0900 [warn]: 'param1' parameter is deprecated: Use new_param1 instead
2016-09-07 15:50:16 +0900 [error]: config error file="test.conf" error="'param2' parameter is already removed: Use new_param2 instead"

If you want to remove existing parameter, use following step:

  • Add deprecated to parameter and release new version
  • Release several minor versions
  • Add obosoleted to parameter and release new version
  • Relese new major version and remove parameter


Lastly, v0.12.29 docker image has also been available on Docker Hub.


Happy logging!

Read More

Fluentd v0.14.3 and v0.14.4 has been released

Hi users!

At the end of August, we have released two versions of Fluentd v0.14. Earlier one, v0.14.3, is just to fix dependency problem, without any other fixes. And then, v0.14.4 was released with many bug fixes and improvements, including very important fixes for Windows environment.

Here are major changes (full ChangeLog is available here):

Core: Improved process management on Windows

The previous versions of Fluentd v0.14 had some known issues about process management on Windows environment. Fluentd did unix-like signal based process control, but it doesn't work well enough on Windows environment. To solve this problem, we introduced a new process control method using pipes in ServerEngine, and fixed Fluentd to depend on it.

On this release, we fixed all known issues about Windows environment!

Core: Filter API improvements

We added a new method for Filter plugins, so now Filter plugin has two methods to filter each records.

  • #filter(tag, time, record) => record: This method is expected to return (updated) record, or nil to drop that record
  • #filter_with_time(tag, time, record) => time, record: This method is expected to return time and record to update time, in addition to record

Core: deprecated/obsoleted options for configuration parameter

Two new options for config_param has been introduced. These options are to show specified parameter is deprecated or obsoleted to users in logs explicitly. These options make it easy to update plugins for plugin authors.

config_param :myparam0, :string, obsoleted: "This parameter was removed at v0.8.9. Use myparam2 instead."
config_param :myparam1, :string, deprecated: "This parameter will be removed at next major version. Use myparam2 instead."

Parameters marked as obsoleted are outdated, and raise errors if users configure Fluentd to use these parameters. On the other hand, deprecated parameter can be used, but it's not recommended to use it. Fluentd shows warning for the use of deprecated parameters.

If you want to remove any configuration parameters, release a version with deprecated options on the parameters to be removed. And then, update options to use obsoleted on that parameters in following release. Users can know which parameters are not good to use, and plugin authors can make guides for users how to migrate their configurations.

in_forward: Adding source hostname into records

The new parameter source_hostname_key has been introduced in forward input plugin. This parameter is useful when your forwarders hard to add hostname or comparing hostname for debugging, See this comment.

When this parameter is specified, in_forward iterates / mutates incoming events and it needs additional processing resource. If your environment is very high traffic, check the performance first.

(This parameter is also introduced in Fluentd v0.12.28.)

Major bug fixes

  • fix not to continue to restart worker processes for configuration errors #1183
  • fix not to start output enqueue/flush threads until plugin completes to start itself #1190
  • fix a bug to miss to set umask #1152
  • fix to start plugin instances under MultiOutput (e.g., copy plugins) #1167

Enjoy logging!

Read More

Fluentd v0.12.28 has been released

Hi users!

We have released Fluentd version 0.12.28. Here are the changes:

New features / Enhancement

  • in_forward: Add source_hostname_key parameter: #807

Bug fixes

  • parser: Ensure strict mode is used for Oj parser: #1147

in_forward: Add source_hostname_key parameter

This parameter is useful when your forwarders hard to add hostname or comparing hostname for debugging, See this comment.

When this parameter is specified, in_forward iterates / mutates incoming events and it needs additional processing resource. If your environment is very high traffic, check the performance first.

parser: Ensure strict mode is used for Oj parser

Oj parser treats : started string as a symbol, not string. It causes plugins raise an error like "undefined method `strip' for #<Symbol:0x000000033f370c>".

v0.12.28 fixes this problem.


Lastly, v0.12.28 docker image has also been available on Docker Hub.


Happy logging!

Read More

About Fluentd

Fluentd is an open source data collector to simplify log management.

Learn

Want to learn more about Fluentd? Check out these pages.

Follow Us!