Fluentd Blog

Fluentd v1.7.2 has been released

Hi users!

We have released v1.7.2. ChangeLog is here. This release includes several fixes.

in_tcp: Add security/client to restrict access

This is similar to in_forward. You can limit the client by host/network. Here is configuration example:

<source>
  @type tcp
  # other parameters...
  <security>
    <client>
      host app_server1.com
    </client>
    <client>
      host web_server1.com
    </client>
  </security>
</source>

With this configuration, in_tcp rejects the data from other hosts, e.g. app_server2.com.

Major bug fixes

• buf_file/buf_file_single: fix to handle compress data during restart
• plugin: Use __send__ to avoid conflict with user defined send method

Enjoy logging!

Read More

Fluentd v1.7.1 has been released

Hi users!

We have released v1.7.1. ChangeLog is here. This release includes several enhancements and fixes.

parser_syslog: Add string parser for rfc3164 message

v1.7.1 introduces parser_type parameter to choose internal implementation for rfc3164 message.

parser_syslog uses regexp parser to parse rfc3164 message by default but it doesn't work for some patterns, see this issue. string parser avoids this problem.

Here is a configuration example:

<source>
  @type syslog
  <parse>
    parser_type string
  </buffer>
  # other parameters...
</source>

string parser is 2x faster than regexp parser, so we recommend to use string parser for new deployment.

Major bug fixes

• buf_file/buf_file_single: fix to ignore placeholder based path during resume
• buf_file: Ensure to remove metadata file after buffer creation failure
• server helper: Ignore ETIMEDOUT error in SSL_accept

Enjoy logging!

Read More

Fluentd v1.7.0 has been released

Hi users!

We have released v1.7.0. ChangeLog is here. This release includes new buffer plugin and some improvement.

Add file_single buffer plugin

This new buffer plugin aims to reduce the number of IO operation for high traffic environment. buf_file_single doesn't have .meta file and it embeds the chunk key value to the file path like v0.12's buf_file.

<match pattern>
  @type forward
  <buffer tag>
    @type file_single
    path /path/to/buffer/test
  </buffer>
  # other parameters...
</match>

With this configuration, here is buffer path example with test.log event tag.

# format is /$path_parameter/fsb.$tag.$chunk_id.buf
/path/to/buffer/test/fsb.test.log.b513b61c9791029c2513b61c9791029c2.buf

buf_file_single now has one limitation. chunk keys must be tag or one field key.

<buffer tag>       # OK
<buffer key>       # OK
<buffer time>      # NG
<buffer key1,key2> # NG

We will remove this limitation by adding metadata header in the file.

Add http output plugin

You can now send data to your destination via HTTP/HTTPS.

See out_http article in the document.

formatter_csv/parser_csv: Improve the performance

formatter_csv is now 2x faster by avoiding ruby's CSV object per event.

parser_csv introduces parser_type parameter to change the internal parser. If set parser_type fast, parser_csv uses own fast parser. The drawback of fast parser is not robust unlike ruby's CSV parser. So if you use fast parser with broken CSV, it may generate broken record. You need to check your CSV is correct before use fast parser.

in_tcp: Emit multiple events at a time

Previous in_tcp emits events one by one even when accepts multiple events in received data. in_tcp now emits multiple events at a time. This reduces CPU usage because emit is heavy task.

core: Improve several points

• Fluent::EventTime.now uses 2x faster implementation
• Remove unncessary data from buffer/output
• Avoid unnessary object allocation

This may reduce CPU usage, a few percent.

Major bug fixes

• output: Fix data lost on decompression by EOF check bug
• in_tail: Don't call parser's configure twice. This fixes the regression of configuration error.

Enjoy logging!

Read More

Fluentd v1.6.3 has been released

Hi users!

We have released v1.6.3. ChangeLog is here. These release includes several enhancements.

in_syslog: Add emit_unmatched_lines parameter

This parameter enables you to get unparseable log as an event. This is same as in_tail's one.

If true, emitted event is like below:

tag: syslog.unmatched  # 'syslog' depends on tag parameter
time: parsed time
record: {"unmatched_line":"broken line"}

buf_file: Add path_suffix parameter

You can change the suffix of buffer file. Here is an example:

# default
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta
# with 'path_suffix .buf'
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf.meta

This parameter is useful when .log suffix is not fit for your system,

Bug fixes

• http_server helper: Fix method re-define warning

Enjoy logging!

Read More

Fluentd v1.6.1 and v1.6.2 has been released

Hi users!

We have released v1.6.1 and v1.6.2. ChangeLog is here. These release includes several enhancements and bug fixes.

http_server helper: Add title argument to the API

This fix is important to launch HTTP servers in different plugins. Old implementation uses same thread identifier for all HTTP servers and it won't work.

Check also API article.

certificates: Support all private keys OpenSSL supports

fluentd assumes private key is RSA before. Now, other types are supported, e.g. EC and DSA.

Other bug fixes

• pluginconfigformatter: use new doc URL
• out_forward: Avoid zero division error when there are no available nodese

Enjoy logging!

Read More