Fluentd v0.14.15 has been released

Hi users!

We have released v0.14.15. Here are the changes (full ChangeLog is available here):

We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.

<worker N> directive

We introduced built-in multi-process feature since v0.14.12(See release announcement). New <worker N> directive enhances multi-process feature.

The limitation of multi-process was non-multi-process ready plugin can't be used together. For example, we can't use in_tail plugin with workers parameter. Following configuration raises an error at startup.

<system>
  workers 4
<system>
<source>
  @type forward
</source>
<source>
  @type tail
</source>

<worker N> directive resolves this problem. You can invoke any plugins in specific worker(N is worker index) and plugins under <worker N> acts as if they work on single process environment. We can mix multi-process ready and non-multi-process ready plugins in one instance. Working example with above configuration is like below:

<system>
  workers 4
<system>
# work on multi process environment. worker0 - worker3 invoke in_forward plugin
<source>
  @type forward
</source>
# work on only worker 0. worker1 - worker3 don't invoke in_tail plugin
<worker 0>
  <source>
    @type tail
  </source>
</worker>
# <worker 1>, <worker 2> or <worker 3> is also ok

Plugin: Add formatted_to_msgpack_binary? method to Output plugin API

We use formatted_to_msgpack_binary method to tell "This plugin uses msgpack for event serialization" to output core when you customize format method. This name is not rubyish, so we add formatted_to_msgpack_binary? method for better name. We recommend to use formatted_to_msgpack_binary? for new plugins.

Of course, formatted_to_msgpack_binary method is available for existing plugins.

Windows: Add several options to configure Fluentd windows service information

By default, Fluentd uses following settings for Windows service registration:

  • Service name: fluentdwinsvc
  • Display name: Fluentd Windows Service
  • Service description: Fluentd is an event collector system.

Since v0.14.15, you can configure these settings via command line options.

  • Service name: --winsvc-name NAME
  • Display name: --winsvc-display-name DISPLAY_NAME
  • Service description: --winsvc-desc DESC

Major bug fixes

  • in_http: Fix X-Forwarded-For header handling. Accpet multiple headers
  • windows: Quote the file path to the Ruby bin directory when starting fluentd as a windows service

Enjoy logging!

Read More

Fluentd v0.14.14 has been released

Hi users!

We have released v0.14.14 with several bug fixes and plugin improvement.

Here are the changes (full ChangeLog is available here):

We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.

in_monitor_agent: Start one HTTP server per worker on sequential port numbers

in_monitor_agent now launches HTTP server per worker on multiprocess environment. The port number is based on port parameter and worker id. For example, if you have following configuration:

<system>
  workers 4
<system>
<source>
  @type monitor_agent
  port 25000
</source>

worker 1's port is 25000, worker 2's port is 25001 and worker 4's port is 25003.

in_http: Support application/msgpack header

fluent-bit has out_http and it uses application/json or application/msgpack header in the request. fluentd's in_http now accepts application/msgpack request from fluent-bit :)

filter_parser: Add emit_invalid_record_to_error parameter

By default, filter_parser emits invalid record to @ERROR label when parsing failed. This is useful for rescuing invalid records in your data pipeline. But users sometimes want to ignore such records for mixed logs. You can now ignore unparseable records by setting emit_invalid_record_to_error false in the configuration.

<filter pattern>
  @type parser
  # ignore non-json logs
  emit_invalid_record_to_error false
  <parse>
    @format json
  </parse>
</filter>

Major bug fixes

  • config: Set encoding forcefully to avoid UndefinedConversionError
  • config: Fix log_level handling in
  • Fix Input and Output deadlock when buffer is full during startup

Enjoy logging!

Read More

Fluentd v0.12.33 has been released

Hi users!

We have released Fluentd version 0.12.33. Here are the changes:

New features / Enhancement

Bug fixes

parser_syslog: Support RFC5424 format

parser_syslog now supports RFC5424 format. Here is the result of RFC5424 format message:

# Incoming message
<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut="3" eventSource="Application" eventID="11211"] Hi, from Fluentd!
# Parsed result
{"pri" => 16, "host" => "192.168.0.1", "ident" => "fluentd", "pid" => "11111", "msgid" => "ID24224", "extradata" => "[exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"]", "message" => "Hi, from Fluentd!"}

By default, parser_syslog assumes incoming message is RFC3614 format. You can parse RFC5424 format by setting message_format rfc5424 in parser configuration.
In addition, you can use message_format auto to handle two message formats in one stream. parser_syslog detects message format automatically and parse it.

Of course, you can use this feature in in_syslog because in_syslog uses parser_syslog internally.

<source>
  @type syslog
  @id in_syslog
  message_format rfc5424
  tag system.syslog
</source>

in_tail: Add 2 features

We added limit_recently_modified parameter. This is useful when use * in the path. If target files are not updated within limit_recently_modified, such files are ignored from watching list.

Another one is adding skip_refresh_on_startup parameter. By default, in_tail reads file content during startup when you set read_from_head true or after restarted.
The problem is this is done by main thread, so it blocks other startup routines. This is no problem when unread lines are small. But it is huge, consuming lines takes long time. skip_refresh_on_startup parameter delays the watching list update so useful for above case.


Lastly, v0.12.33 docker image has also been available on Docker Hub.


Happy logging!

Read More

Fluentd v0.14.13 has been released

Hi users!

We have released v0.14.13 with several bug fixes and new utilities.

Here are the changes (full ChangeLog is available here):

We say again, fluentd v0.14 is still development version. We need your feedback seriously! If you try to use v0.14, check your configuration and plugins carefully.

Add 'fluent-plugin-generate' command

fluent-plugin-generate helps you to create new plugin for v0.14. It generates skelton code and rubygems files. Here is the step for new plugin release:

# Create plugin files. generate command takes two arguments, plugin type and name
% fluent-plugin-generate output test
License: Apache-2.0
create Gemfile
create README.md
create Rakefile
create fluent-plugin-test.gemspec
create lib/fluent/plugin/out_test.rb
create test/helper.rb
create test/plugin/test_out_test.rb
Initialized empty Git repository in /Users/repeatedly/tmp/fluent-plugin-test/.git/

# Set up develop environment
% gem install bundler # if you don't install bundler yet
% edit fluent-plugin-test.gemspec # need to update summary, description and homepage to use bundle
% bundle install --path vendor/bundle
% bundle exec rake test # check running test

# edit plugin code and test
% edit lib/fluent/plugin/out_test.rb
% edit test/plugin/test_out_test.rb

# Update README and publish gem
% edit README.md
% bundle exec rake release

Currently, our plugin development article for v0.14 is incomplete. We will write new article with v0.14 API soon.

Skip record with Output#format API

In v0.12, Output plugin can skip invalid / unexpected record by using Fluent::BufferedOutput#format_stream API. But v0.14 API doesn't provide it so hard to migrate v0.12 plugin into v0.14.

To avoid this problem, Fluent::Plugin::Output#format API can return nil to skip the record. Here is an example:

helpers :event_emitter # need this code to use router in Output plugin

def format(tag, time, record)
  if record['field'] == 'foo'
    record.to_json + "\n"
  else
    router.emit_error_event(tag, time, record, ArgumentError.new("received event is invalid in xxx plugin"))
    nil
  end
end

With above code, if the value of field is 'foo', its events are stored into buffer. On the other hand, other events are routed to error stream and sucn events are not stored in buffer.

Major bug fixes

  • output: Secondary calculation should consider retry_max_times
  • parser: Fix missing parser_regex require
  • in_tail: Untracked files should be removed from watching list to avoid memory bloat
  • record_transformer: Revert "Use BasicObject for cleanroom" for enable_ruby regression.

Enjoy logging!

Read More

Debian based fluentd docker image has been released

Hi users!

We officially provide Alpine Linux based Fluentd docker image. Alpine Linux is light weight and this is good for Fluentd use cases. But some plugins, e.g. fluent-plugin-systemd, don't work on Alpine Linux, so we received the request "Could you provide other OS based image for xxx."

To resolve this problem, @tyranron works on improving Docker image management. Easy to add other OS, more automated release and tagging, build testing and more. Thanks to @tyranron for your hard work :)

In this result, we start to provide Debian based docker image together.

Debian images by tyranron · Pull Request #71 · fluent/fluentd-docker-image

The default is still Alpine Linux but you can choose Debian version for your requirement. See Supported tags section in README for all available images.

If you have any problem, please let me know.


Happy logging!

Read More

About Fluentd

Fluentd is an open source data collector to simplify log management.

Learn

Want to learn more about Fluentd? Check out these pages.

Follow Us!