Fluentd Blog

Fluentd v1.7.0 has been released

Hi users!

We have released v1.7.0. ChangeLog is here. This release includes new buffer plugin and some improvement.

Add file_single buffer plugin

This new buffer plugin aims to reduce the number of IO operation for high traffic environment. buf_file_single doesn't have .meta file and it embeds the chunk key value to the file path like v0.12's buf_file.

<match pattern>
  @type forward
  <buffer tag>
    @type file_single
    path /path/to/buffer/test
  </buffer>
  # other parameters...
</match>

With this configuration, here is buffer path example with test.log event tag.

# format is /$path_parameter/fsb.$tag.$chunk_id.buf
/path/to/buffer/test/fsb.test.log.b513b61c9791029c2513b61c9791029c2.buf

buf_file_single now has one limitation. chunk keys must be tag or one field key.

<buffer tag>       # OK
<buffer key>       # OK
<buffer time>      # NG
<buffer key1,key2> # NG

We will remove this limitation by adding metadata header in the file.

Add http output plugin

You can now send data to your destination via HTTP/HTTPS.

See out_http article in the document.

formatter_csv/parser_csv: Improve the performance

formatter_csv is now 2x faster by avoiding ruby's CSV object per event.

parser_csv introduces parser_type parameter to change the internal parser. If set parser_type fast, parser_csv uses own fast parser. The drawback of fast parser is not robust unlike ruby's CSV parser. So if you use fast parser with broken CSV, it may generate broken record. You need to check your CSV is correct before use fast parser.

in_tcp: Emit multiple events at a time

Previous in_tcp emits events one by one even when accepts multiple events in received data. in_tcp now emits multiple events at a time. This reduces CPU usage because emit is heavy task.

core: Improve several points

• Fluent::EventTime.now uses 2x faster implementation
• Remove unncessary data from buffer/output
• Avoid unnessary object allocation

This may reduce CPU usage, a few percent.

Major bug fixes

• output: Fix data lost on decompression by EOF check bug
• in_tail: Don't call parser's configure twice. This fixes the regression of configuration error.

Enjoy logging!