Fluentd v0.14.19 has been released

Hi users!

We have released v0.14.19. Here are the changes (full ChangeLog is available here): This release is mainly for bug fix and several plugin improvement.

We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.

filter_grep: New <regexp>/<exclude> section

We introduce <regexp>/<exclude> section to filter_grep. Here is an example:

<filter pattern>
  @type grep
  <regexp>
    key message
    pattern cool
  </regexp>
  <regexp>
    key hostname
    pattern ^web\d+\.example\.com$
  </regexp>
</filter>

This is same as:

<filter pattern>
  @type grep
  regexp1 message cool
  regexp2 hostname ^web\d+\.example\.com$
</filter>

New section approach is easy to parse by fluentd core and we have a plan to add nested record access support via new section parameters.

in_syslog improvement

  • Allow more characters in identifier part

Old implementation allows only [a-zA-Z0-9_\/\.\-] characters in identifier part. This follows RFC3614 but real world systems use more characters. For the real world, in_syslog now accept more characters except , : and [.

  • Add resolve_hostname parameter

This parameter is useful for reducing resolve cost when you don't use source_hostname_key.

Major bug fixes

  • output: Fix race condition during retry

Users sometimes hit "undefined method `next_time' for nil:NilClass" error inside flush thread of output plugins. v0.14.19 fixed it.

We also released v0.12.38. This is for adding v0.14.19 features to in_syslog and filter_grep to reduce the differences between v0.12 and v0.14.

Enjoy logging!

Read More

Fluentd v0.14.17 has been released

Hi users!

We have released v0.14.17. Here are the changes (full ChangeLog is available here): This release is mainly for bug fixes.

We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.

server: Accept private key for TLS server without passphrase

The private_key_passphrase and ca_private_key_passphrase for TLS setting are now optional. It means you can use private key without passphrase in server plugin helper based plugins, e.g. in_forward/out_forward.

Major bug fixes

  • out_forward: Fix ack handling
  • config: Mask all secret parameters in worker section

This version fixed out_forward's require_ack_response regressions. If you use require_ack_response with v0.14, we recommend to update fluentd v0.14 to v0.14.17.

Enjoy logging!

Read More

Fluentd v0.14.16 has been released

Hi users!

We have released v0.14.16. Here are the changes (full ChangeLog is available here): This release is mainly for fixing regressions.

We say again, fluentd v0.14 is still development version. We need your feedback! If you try v0.14, check your configuration and plugins carefully.

Allow null character in double-quoted string

You can use null character \0 with double-quoted string. This is useful for delimiter, e.g. tcp input, tsv parser, etc. Here is an example:

<source>
  @type tail
  path /path/to/app.log
  tag tail.test
  format tsv
  keys a,b,c
  delimiter "\0" # \0 or '\0' doesn't work
</source>

Major bug fixes

  • out_forward: Call proper method for each connection type
  • in_monitor_agent: check variable buffer is a Buffer instance

Enjoy logging!

Read More

Fluentd v0.14.15 has been released

Hi users!

We have released v0.14.15. Here are the changes (full ChangeLog is available here):

We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.

<worker N> directive

We introduced built-in multi-process feature since v0.14.12(See release announcement). New <worker N> directive enhances multi-process feature.

The limitation of multi-process was non-multi-process ready plugin can't be used together. For example, we can't use in_tail plugin with workers parameter. Following configuration raises an error at startup.

<system>
  workers 4
<system>
<source>
  @type forward
</source>
<source>
  @type tail
</source>

<worker N> directive resolves this problem. You can invoke any plugins in specific worker(N is worker index) and plugins under <worker N> acts as if they work on single process environment. We can mix multi-process ready and non-multi-process ready plugins in one instance. Working example with above configuration is like below:

<system>
  workers 4
<system>
# work on multi process environment. worker0 - worker3 invoke in_forward plugin
<source>
  @type forward
</source>
# work on only worker 0. worker1 - worker3 don't invoke in_tail plugin
<worker 0>
  <source>
    @type tail
  </source>
</worker>
# <worker 1>, <worker 2> or <worker 3> is also ok

Plugin: Add formatted_to_msgpack_binary? method to Output plugin API

We use formatted_to_msgpack_binary method to tell "This plugin uses msgpack for event serialization" to output core when you customize format method. This name is not rubyish, so we add formatted_to_msgpack_binary? method for better name. We recommend to use formatted_to_msgpack_binary? for new plugins.

Of course, formatted_to_msgpack_binary method is available for existing plugins.

Windows: Add several options to configure Fluentd windows service information

By default, Fluentd uses following settings for Windows service registration:

  • Service name: fluentdwinsvc
  • Display name: Fluentd Windows Service
  • Service description: Fluentd is an event collector system.

Since v0.14.15, you can configure these settings via command line options.

  • Service name: --winsvc-name NAME
  • Display name: --winsvc-display-name DISPLAY_NAME
  • Service description: --winsvc-desc DESC

Major bug fixes

  • in_http: Fix X-Forwarded-For header handling. Accpet multiple headers
  • windows: Quote the file path to the Ruby bin directory when starting fluentd as a windows service

Enjoy logging!

Read More

Fluentd v0.14.14 has been released

Hi users!

We have released v0.14.14 with several bug fixes and plugin improvement.

Here are the changes (full ChangeLog is available here):

We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.

in_monitor_agent: Start one HTTP server per worker on sequential port numbers

in_monitor_agent now launches HTTP server per worker on multiprocess environment. The port number is based on port parameter and worker id. For example, if you have following configuration:

<system>
  workers 4
<system>
<source>
  @type monitor_agent
  port 25000
</source>

worker 1's port is 25000, worker 2's port is 25001 and worker 4's port is 25003.

in_http: Support application/msgpack header

fluent-bit has out_http and it uses application/json or application/msgpack header in the request. fluentd's in_http now accepts application/msgpack request from fluent-bit :)

filter_parser: Add emit_invalid_record_to_error parameter

By default, filter_parser emits invalid record to @ERROR label when parsing failed. This is useful for rescuing invalid records in your data pipeline. But users sometimes want to ignore such records for mixed logs. You can now ignore unparseable records by setting emit_invalid_record_to_error false in the configuration.

<filter pattern>
  @type parser
  # ignore non-json logs
  emit_invalid_record_to_error false
  <parse>
    @format json
  </parse>
</filter>

Major bug fixes

  • config: Set encoding forcefully to avoid UndefinedConversionError
  • config: Fix log_level handling in
  • Fix Input and Output deadlock when buffer is full during startup

Enjoy logging!

Read More

About Fluentd

Fluentd is an open source data collector to simplify log management.

Learn

Want to learn more about Fluentd? Check out these pages.

Follow Us!