Fluentd Blog

Fluentd v0.12.3 is released

Hi users!

We have just released Fluentd version 0.12.3. Here are the changes:

New features / Enhancement

• parser: nginx and apache2 format can now parse access log without http-version: #535
• filter_record_transformer: Allow non-string values like array / hash: #530
• plugin: Add writable check for path related parameter: #401
• command: -g and -G now works in embedded Ruby environment like Treasure Agent: #423

Bug fixes

• buffer: Fix to flush a buffer by USR1 signal even on retrying: #531
• buffer: Fix TimeSlicedOutput doesn't flush with SIGUSR1: #533
• buffer: Fix broken calc_retry_wait if Integer is used for retry_wait parameter: #529
• engine: Fix SIGHUP does not reload config: #537

This release is mainly for improving stability.

Support array / hash values in record_transformer filter

In old record_transfomer filter, values in <record> becomes string value. For example, if you have following configuration,

<filter debug.**>
  type record_transformer
  <record>
    hash {  
      "hostname": "${hostname}",
      "tag": "${tag}"
    }
  </record>
</filter>
<match debug.**>
  type stdout
</match>

We got a stringized result, not hash.

2015-01-16 00:10:20 +0900 debug.test: {"k":"v","hash":"{\"hostname\":\"server1\",\"tag\":\"debug.test\"}"}

Since this version, got hash value like below.

2015-01-16 00:10:20 +0900 debug.test: {"k":"v","hash":{"hostname":"server1","tag":"debug.test"}}

Improve buffer related signal handling

In old version, fluentd waits retry wait even if received USR1 signal. From now, fluentd flushes buffers immediately.

In addition, TimeSlicedOutput plugin, e.g. out_file, also flushes buffers by USR1.

Happy logging!

Read More

Fluentd v0.10.58 is released

Hi users!

We have just released Fluentd version 0.10.58. Here are the changes:

• parser/formatter: Add base class and Plugin.new_xxx / Plugin.register_xxx APIs: #495,#497
• parser: Fix blank column handling of TSVParser: #506
• formatter: Add new CSV formatter: #489
• formatter: Add msgpack format to built-in Formatter to dump records: #479
• config: Add self.name to configure_proxy error message: #490
• config: Fix system config using double memory: #496
• config: Fix v1 config to support multiple tag match: #478
• config: Fix Config.bool_value regression for nil value: #477
• input: Add in_dummy plugin: #484
• in_http: Add respond_with_empty_img parameter to return empty gif image: #480
• in_http: Add cors_allow_origins parameter to support CORS request: #473
• buffer: Prevent an exception with large num_retries: #502
• out_file: Don't create world writable directory in daemon mode: #483
• Add router.emit and router= for compatibility with v0.12: #510
• Use router.emit instead of Engine.emit: #511

This release is mainly for improving compatibility with v0.12.

New Parser and Formatter API

In v0.10.57 or earlier, we use following APIs to implement Parser class.

# Parser implementation
module Fluent
  class TextParser
    # Not inherit base class
    class FooParser
      include Configurable

      # Overwrite initialize, configure and parse method

      # register this parser
      TextParser.register_template('foo', Proc.new { FooParser.new })
    end
  end
end

# Instantiate parser
parser = TextParser.new
parser.configure(conf)

New APIs are below:

# Parser implementation
module Fluent
  class TextParser
    # inherit Parser base class
    class FooParser < Parser
      # register this plugin without Proc.new trick
      Plugin.register_parser('foo', self)

      # Overwrite initialize, configure and parse method
    end
  end
end

# Instantiate parser
parser = Plugin.new_parser('foo')
parser.configure(conf)

Formatter also has same APIs. New API is consistent with Input / Output plugins. We still have old API but we recommend to use new API for new parser / formatter plugins.

Add router attribute to Input / Output

In v0.12, Input / Output plugins have router attribute for Label and Engine.emit is deprecated(Engine.emit forwards events to root_agent.emit).

v0.10.58 adds router attribute to Input / Output plugins for API compatibility with v0.12. Changing Engine.emit to router.emit has no effect on v0.10 but such plugin works on both v0.10 and v0.12 properly.

Happy logging!

Read More

Fluentd v0.12 is Released

We are pleased to announce that we have released Fluentd v0.12 today. v0.12 introduces nice features and improves internal structures (in anticipation for v1).

RubyGems: fluentd 0.12.0

New Features

Configuration

--use-v1-config is used by default, which means that the user can use JSON arrays/hashes and embedded Ruby code natively. See the docs for further details.

In addition, we introduced the @ prefix for system parameter or label symbols to avoid conflicts with plugin parameters.

• @type specifies plugin type
• @id specifies plugin id. in_monitor_agent uses this value for plugin_id field.
• @label specifies label. See the Label section below.

type and id are still valid to ensure backward compatibility.

A Small Syntactical Incompatibility

v1 config has one incompatibility problem: if the value of a parameter starts with the characters '{' or '[', it is now interpreted as JSON. To keep using the existing config, the parameter value must be explicitly quoted. Check "Multi line support for array and hash values" for further details.

Filter

v0.12 introduces filter plugins to process event streams. With filter plugins, the old trick of "using output plugins as filter" is no longer needed.

Seeing is believing:

<filter pattern1>
  @type filter_foo
  # ...
</filter>
<filter pattern2>
  @type filter_bar
  # ...
</filter>

Fluentd tries to apply a filter chain to event streams. If a tag is matched with pattern1 and pattern2, Fluentd applies filter_foo and filter_bar top-to-bottom (filter_foo followed by filter_bar). See this v0.12 configuration as a detailed example.

v0.12 ships with grep and record_transformer plugins.

• grep

The grep filter is a filter version of fluent-plugin-grep of output plugin. The configuration example is below:

<filter foo.**>
  @type grep
  regexp1 message INFO
</filter>

If the value of the "message" field doesn't match "INFO", such events are removed from event stream. grep filter doesn't support the old input_key and regexp pair.

• record_transformer

The record_transformer filter is a filter version of fluent-plugin-record-reformer output plugin.

<filter bar.**>
  @type record_transformer
  <record>
    hostname ${hostname}
  </record>
</filter>

Filters don't allow tag rewriting. So, unlike record_reformer, record_transformer doesn't have tag-related parameters.

Label

Label is useful for grouping Filter and Output Plugins. Labels simplify event stream routing and reduces the complexity of configuration files.

Again, seeing is believing:

# The events coming from this source are routed to the file <match>.
<source>
  @type forward
  port 24224
</source>

# The events from this source are routed to s3 <match> in <label @STAGING>
<source>
  @type forward
  @label @STAGING
  port 24225
</source>

<match forward.**>
  @type file
  # ...
</match>

# Label symbol should have `@` prefix bacause it is not a pattern.
<label @STAGING>
  <match forward.**>
    @type s3
    # ...
  </match>
</label>

At-Least-Once Semantics in Forward Plugins

Fluentd now supports at-least-once semantics between in_forward and out_forward. If you set require_ack_response parameter in out_forward plugin, out_forward waits ack response from in_forward.

<match forward.**>
  @type forward
  # ...
  require_ack_response true
  ack_response_timeout 60s # default is 190s
  <server>
    # ...
  </server>
</match>

Give it a Whirl

We tested v0.12 in some high-load environments without issues. That said, if you run into trouble with v0.12 in your environment, please file an issue on GitHub.

In addition, we are now updating Fluentd document for v0.12. More detailed information, e.g. how to write filter plugins, will be available soon!

Enjoy Fluentd!

Read More

Fluentd v0.10.57 is released

Hi users!

We have just released Fluentd version 0.10.57. Here are the changes:

• buffer: Make total_queued_chunk_size thread-safe to avoid race condition with in_monitor_agent: #469
• Add --no-supervisor option to disable the supervisor for supervision tools like runit: #449

This release is for bug fix.

--no-supervisor option

Fluentd has a own supervisor to manage actual worker (See Process Model). It is good for standalone daemon but users might use favorite management tool like runit. --no-supervisor is useful for such cases. If you pass --no-supervisor, Fluentd runs without a supervisor.

• with supervisor

  % ps w -C ruby -C fluentd --no-heading
  23197 pts/1 Sl+ 0:00 /path/to/ruby /path/to/fluentd -c test.conf
  23223 pts/1 Sl+ 0:00 /path/to/ruby /path/to/fluentd -c test.conf

• without supervisor

  % ps w -C ruby -C fluentd --no-heading
  23234 pts/1 Sl+ 0:00 /path/to/ruby /path/to/fluentd -c test.conf --no-supervisor

Happy logging!

Read More

Fluentd Goes Gopher!

I am pleased to announce that Treasure Data just open sourced a lightweight Fluentd forwarder written in Go. Creatively christened as Fluentd Forwarder, it was designed and written with the following goals in mind.

1. A simple forwarder for simple use cases: Fluentd is very versatile and extendable, but sometimes you need something simple. Fluentd Forwarder is meant to be an alternative for such cases.
2. Support for Windows: Yes, Fluentd has an experimental Windows branch in active development, but some community members expressed that they wanted something simpler and less experimental that runs in a Windows environment. Thanks to Go's cross-compilation, Fluentd Forwarder should work well out of the box on Windows.
3. Go is a good fit: For a program written in Ruby, Fluentd is pretty fast and has a tiny memory footprint at 40MB. Also, it has leveraged Ruby's flexibility and popularity to build a rich plugin ecosystem. That said, Go was designed to write efficient middleware productively, and this project was a natural fit in that regard. Plus, Go, like Fluentd, has a cute logo.

Here are some of the salient features of the first release.

1. TCP input and output: By default, Fluentd Forwarder accepts data on 127.0.0.1:24224 and forwards it to 127.0.0.1:24225. You can think of this as in_forward and out_forward.
2. Treasure Data output: Because this project was conceived to help Treasure Data's customers collect data on Windows servers, it also support Treasure Data as an output option.

It is just as important to mention what Fluentd Forwarder does NOT have.

1. Fluentd Forwarder does not have a pluggable architecture, at least not yet. At the moment, the output destination (either TCP or Treasure Data) is determined by a command line option or configuration parameter URL.
2. Also, the forwarding logic is not as sophisticated as that of out_forward.
3. Fluentd Forwarder does not have downloadable binaries yet, although we plan to make it available soon. Right now, you have to get a Go runtime (v1.3 and above) and compile it yourself.

This project is still in its infancy. If you have any feedback, bug report, questions or PULL REQUESTS, they are all welcome!

Finally, many thanks to Moriyoshi Koizumi for writing the initial version. I am excited to see where we go with this sibling project =)

Read More

Fluentd v0.10.56 is released

Hi users!

We have just released Fluentd version 0.10.56. Here are the changes:

• config: Add Fluent::Config::Section#object_id to avoid NoMethodError by object dump tools: #460
• mixin: Fix SetTagKeyMixin and SetTimeKeyMixin which ignores config_set_default since 0.10.53: #462
• input/output: Support 'timezone' configuration parameter (e.g. "+09:00" for JST): #430
• log: Use plugin_id instead of object_id in log message for consistency with in_monitor_agent: #459

This release fixes some bugs and adds timezone support.

timezone parameter

Since v0.10.56, Fluentd provides timezone parameter to specify arbitary timezone, not only utc or localtime. Support formats are offset +09:00 or long name Asia/Tokyo. Short name like JST is not supported.

<match pattern>
  type file
  path /var/log/fluent/myapp
  time_slice_format %Y%m%d
  time_slice_wait 10m
  time_format %Y%m%dT%H%M%S%z
  timezone +08:00
</match>

This is a configuration example. You can use timzeone parameter in time related plugins, e.g. file, formatter and etc. For example, you can see following content in the file.

2014-10-30T19:00:00+08:00       test.foo        {"message":"Hi!"}

We will add timezone document to plugin articles later.

Happy logging!

Read More

Fluentd Will Be at RubyConf 2014!

Great news: Fluentd will be at RubyConf 2014 next month! In fact, if we count our sister project Norikra, there will be not only one but TWO Fluentd-related talks at RubyConf =)

Many of Fluentd's core contributors will be at the event. In addition to the two speakers @tagomoris and @kiyoto, the main maintainer @repeatedly will be hanging around to answer your questions.

Happy log collecting, and we look forward to seeing you in San Diego!

Read More

Fluentd v0.10.55 is released

Hi users!

We have just released Fluentd version 0.10.55. We forgot cherry-picking one commit so version 0.10.54 is skipped.
Here are the changes:

• config: Change v1's non-quoted string literal behaviour to v0's behaviour: #436
• config: Relax '@' handling in the plugin configuration for backward compatibility: #455
• config: Add --use-v0-config for keeping old configuration: #434
• config: Add single quoted literal in v1 configuration: #437
• config: Improve error message of Array / Hash parse error: #457
• input: Reduce shutdown time in some network related plugins when use Cool.io 1.2: #440
• parser: Use ParserError instead of general exception classes: #453
• Remove unused zfile buffer plugin: #445

This release improves and changes v1 configuration.

Change v1's non-quoted string literal behaviour

Old v1's non-quoted string literal is same as double-quoted string literal so v1-config user should escape some characters, e.g. \, with \. It has a problem which hard to write regular expression and hard to migrate from v0-config. After several discussion, we decided to change v1's non-quoted string literal behaviour to v0's non-quoted string. You don't need \ escape in the non-quoted string.

In this result, you can easy to migrate from v0 to v1 configuration. This is good for existence Fluentd and td-agent users. We will add each string literal explanation to configuration article on Fluentd document.

This change breaks backward compatibility of --use-v1-config. If you already use --use-v1-config or td-agent 2, see this thread for avoiding incompatible problem: https://groups.google.com/forum/#!topic/fluentd/4C36iAyI87E

Add --use-v0-config option

Fluentd v0.12 or later uses v1 configuration by default. If you want to keep v0 configuration, this option is useful.

Happy logging!

Read More

Fluentd v0.10.53 is released

Hi users!

We have just released Fluentd version 0.10.53.
Here are the changes:

• in_tail: Fix forget to detach Closer timer object: #394
• in_debug_agent: Fix config_param type of unix_path: #392
• in_syslog: Add includesourcehost and sourcehostKey options: #391
• config: Dump V1 escaped parameter correctly at start phase: #389
• config: Fix bug for config_param of bool value without defaults: #410
• config: Add 'alias' option to config_section and config_param: #402
• buffer: Add 'disable_retry_limit' option: #415
• engine: Reduce memory usage when retry_limit is large: #388
• parser: Add apache_error format: #396
• supervisor: Dump Fluentd version at dry run mode: #384
• test: InputTestDriver wait 'run' until input plugin emits all test data: #397

This release fixed several bugs and added several useful options.

apache_error format

Fluentd parser provides apache_error format as a built-in. Some users collect Apache error logs using Fluentd on production, so supporting this format seems useful for other users.

disable_retry_limit option

This is for buffered output plugins. If you set disable_retry_limit to true, output tries to retry forever and <secondary> is never called.

You can simulate same behaviour using very large value with retry_limit but disable_retry_limit is more clear.

Started v0.12 development in master branch

After released v0.10.53, master branch is now for v0.12.
We are now developing Filter and Label feature on this PR.

If you have v0.10 specific problem, please send PR to v0.10 branch or open an issue with v0.10 label.

Happy logging!

Read More

Fluentd-UI: Managing Fluentd from a Browser

Over the last three years, Fluentd has helped various organizations unify their logging infrastructure. As our user base grew, it became clear that diving into a command line to set up the first Fluentd instance can be a bit daunting.

So, we enlisted the help of a badass Rails engineer to create a browser-based admin UI for Fluentd. Pictures are worth a thousand words, so here is what it looks like:

With Fluentd-UI, starting, stopping, configuring, restarting Fluentd is only a few clicks away in your browser.

You can find the code on GitHub, and as always, feel free to ask us questions on our mailing list!

Read More