Fluentd Blog

Fluentd v1.7.3 has been released

Hi users!

We have released v1.7.3. ChangeLog is here. This release is mainly for fixing socket cache bug of out_forward.

in_syslog: Replace priority_key with severity_key

in_syslog's priority_key parameter is misleading name because it sets severity, not priority value. So we add severity_key parameter.

<source>
  @type syslog
  severity_key severity
  tag syslog
</source>

priority_key is still supported for existing users but we will remove priority_key parameter at fluentd v2.

Major bug fixes

• out_forward: Fix nil error after purge obsoleted socket in socket cache

Enjoy logging!

Read More

Fluentd v1.7.2 has been released

Hi users!

We have released v1.7.2. ChangeLog is here. This release includes several fixes.

in_tcp: Add security/client to restrict access

This is similar to in_forward. You can limit the client by host/network. Here is configuration example:

<source>
  @type tcp
  # other parameters...
  <security>
    <client>
      host app_server1.com
    </client>
    <client>
      host web_server1.com
    </client>
  </security>
</source>

With this configuration, in_tcp rejects the data from other hosts, e.g. app_server2.com.

Major bug fixes

• buf_file/buf_file_single: fix to handle compress data during restart
• plugin: Use __send__ to avoid conflict with user defined send method

Enjoy logging!

Read More

Fluentd v1.7.1 has been released

Hi users!

We have released v1.7.1. ChangeLog is here. This release includes several enhancements and fixes.

parser_syslog: Add string parser for rfc3164 message

v1.7.1 introduces parser_type parameter to choose internal implementation for rfc3164 message.

parser_syslog uses regexp parser to parse rfc3164 message by default but it doesn't work for some patterns, see this issue. string parser avoids this problem.

Here is a configuration example:

<source>
  @type syslog
  <parse>
    parser_type string
  </buffer>
  # other parameters...
</source>

string parser is 2x faster than regexp parser, so we recommend to use string parser for new deployment.

Major bug fixes

• buf_file/buf_file_single: fix to ignore placeholder based path during resume
• buf_file: Ensure to remove metadata file after buffer creation failure
• server helper: Ignore ETIMEDOUT error in SSL_accept

Enjoy logging!

Read More

Fluentd v1.7.0 has been released

Hi users!

We have released v1.7.0. ChangeLog is here. This release includes new buffer plugin and some improvement.

Add file_single buffer plugin

This new buffer plugin aims to reduce the number of IO operation for high traffic environment. buf_file_single doesn't have .meta file and it embeds the chunk key value to the file path like v0.12's buf_file.

<match pattern>
  @type forward
  <buffer tag>
    @type file_single
    path /path/to/buffer/test
  </buffer>
  # other parameters...
</match>

With this configuration, here is buffer path example with test.log event tag.

# format is /$path_parameter/fsb.$tag.$chunk_id.buf
/path/to/buffer/test/fsb.test.log.b513b61c9791029c2513b61c9791029c2.buf

buf_file_single now has one limitation. chunk keys must be tag or one field key.

<buffer tag>       # OK
<buffer key>       # OK
<buffer time>      # NG
<buffer key1,key2> # NG

We will remove this limitation by adding metadata header in the file.

Add http output plugin

You can now send data to your destination via HTTP/HTTPS.

See out_http article in the document.

formatter_csv/parser_csv: Improve the performance

formatter_csv is now 2x faster by avoiding ruby's CSV object per event.

parser_csv introduces parser_type parameter to change the internal parser. If set parser_type fast, parser_csv uses own fast parser. The drawback of fast parser is not robust unlike ruby's CSV parser. So if you use fast parser with broken CSV, it may generate broken record. You need to check your CSV is correct before use fast parser.

in_tcp: Emit multiple events at a time

Previous in_tcp emits events one by one even when accepts multiple events in received data. in_tcp now emits multiple events at a time. This reduces CPU usage because emit is heavy task.

core: Improve several points

• Fluent::EventTime.now uses 2x faster implementation
• Remove unncessary data from buffer/output
• Avoid unnessary object allocation

This may reduce CPU usage, a few percent.

Major bug fixes

• output: Fix data lost on decompression by EOF check bug
• in_tail: Don't call parser's configure twice. This fixes the regression of configuration error.

Enjoy logging!

Read More

Fluentd v1.6.3 has been released

Hi users!

We have released v1.6.3. ChangeLog is here. These release includes several enhancements.

in_syslog: Add emit_unmatched_lines parameter

This parameter enables you to get unparseable log as an event. This is same as in_tail's one.

If true, emitted event is like below:

tag: syslog.unmatched  # 'syslog' depends on tag parameter
time: parsed time
record: {"unmatched_line":"broken line"}

buf_file: Add path_suffix parameter

You can change the suffix of buffer file. Here is an example:

# default
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta
# with 'path_suffix .buf'
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf.meta

This parameter is useful when .log suffix is not fit for your system,

Bug fixes

• http_server helper: Fix method re-define warning

Enjoy logging!

Read More