Fluentd Blog

Fluentd v1.17.0 has been released

Hi users!

We have released v1.17.0 on 2024-04-30. ChangeLog is here.

This release is a new release of v1.17 series. In this release, we added some new features for some plugins and fixed bugs of Parser.

Enhancement

in_tail: Add glob_policy option for expanding glob capability of path and exclude_path

In this release, we added a new option glob_policy for in_tail plugin.

In previous versions, we can use only * in glob patterns for path and exclude_path option.

Example:

path /path/to/*
exclude_path ["/path/to/*.gz", "/path/to/*.zip"]

From this version, we can also use [], ?, and {} in glob patterns depending on the glob_policy option.

Example:

path "[0-1][2-3].log"
glob_policy extended

Please see the document and #4401 for more information.

out_http: Support AWS Signature Version 4 authentication

In this release, we added a new option aws_sigv4 for the method setting of out_http plugin.

By using this option, out_http can use AWS Signature Version 4.

For example, this allows out_http to write to Amazon OpenSearch Ingestion.

Please see the document and #4459 for more information.

out_http: Add option to reuse connections

In this release, we add a new option reuse_connections for out_http plugin.

This option will improve throughput of out_http.

Please see the document and #4330 for more information.

in_http: Recognize CSP reports as JSON data

In this release, we make the data type of the request where the Content-Type is application/csp-report be considered JSON by default.

Now, in_http can receive Content Security Policy's report by default.

Please see #4282 for more information.

Bug Fixes

Make sure parser returns hash

The record data in an event of Fluentd must be a hash object.

• Life of a Fluentd event

However, in the previous versions, some parser plugins could return a non-hash object, such as an array. It could cause errors in subsequent processing.

In this release, the following parser plugins have been fixed.

• parser_json
• parser_msgpack

The changes are as follows:

• Make sure to return a hash record.
• Make sure to accept only a hash or an array of hash.

Here are the details for each case.

Example of changed behavior

Config:

<source>
  @type tcp
  tag test.tcp
  <parse>
    @type json
  </parse>
</source>

<match test.**>
  @type stdout
</match>

Send an array data:

$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]

The result before this version:

{datetime} test.tcp: [{"k":"v"},{"k2":"v2"}]

The result after this version:

{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}

Example of resolved error

Config:

<source>
  @type tcp
  tag test.tcp
  <parse>
    @type json
    null_empty_string
  </parse>
</source>

<match test.**>
  @type stdout
</match>

Send an array data:

$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]

The result before this version:

{datetime} [error]: #0 unexpected error on reading data host="xxx" port=xxx error_class=NoMethodError error="undefined method `each_key' for [{\"k\"=>\"v\"}, {\"k2\"=>\"v2\"}]:Array"

The result after this version:

{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}

Remaining problem: filter_parser

In the previous versions, filter_parser could return an array record based on this wrong behavior. From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case. This behavior should improve in the future.

Here is an example.

<source>
  @type sample
  tag test.array
  sample {"message": "[{\"k\":\"v\"}, {\"k2\":\"v2\"}]"}
</source>

<filter test.**>
  @type parser
  key_name message
  <parse>
    @type json
  </parse>
</filter>

<match test.**>
  @type stdout
</match>

The result before this version:

{datetime} test.array: [{"k":"v"},{"k2":"v2"}]

The result after this version:

{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: '[{\"k\":\"v\"}, {\"k2\":\"v2\"}]'" location=nil tag="test.array" time=xxx record={"k2"=>"v2"}
{datetime} test.array: {"k":"v"}

These are the major changes for this release.

In addition, some performance improvements have been included! Please see ChangeLog for details!

Enjoy logging!