Fluentd Blog

Fluentd v1.11.2 has been released

Hi users!

We have released v1.11.2. ChangeLog is here.

Allow regular expression in filter/match tag matching

Regular expression, /pattern/, is now allowed in tag matching:

<source>
  @type forward
</source>

# For a.xxx tag
<filter a.**>
  @type record_transformer
  <record>
    new_key a-started tag record
  </record>
</filter>

# For non a.xxx tag like b.xxx/c.xxx/...
<filter /(?!a\.).*/>
  @type record_transformer
  <record>
    new_key other tag record
  </record>
</filter>

<match **>
  @type stdout
</match>

If you send following events:

% echo '{"msg":"hi"}' | fluent-cat a.foo
% echo '{"msg":"hi"}' | fluent-cat b.foo

the log shows:

2020-08-06 13:40:03.111196000 +0900 a.foo: {"msg":"hi","new_key":"a-started tag record"}
2020-08-06 13:40:09.492380000 +0900 b.foo: {"msg":"hi","new_key":"other tag record"}

Of course, simple data pipeline is important for robust log collection. If you need this feature, check your configuration/data flow first.

in_dummy is renamed in_sample

We recommend to use sample instead of dummy:

<source>
  @type sample
  sample {"hello":"world"}
</source>

# If you use fluentd v1.11.1 or earlier, use following configuration
# Fluentd v2 will remove old configuration support
<source>
  @type dummy
  dummy {"hello":"world"}
</source>

Major bug fixes

• buffer: Fix log message for chunk_limit_records case
• buffer: Fix timekey optimization for non-windows platform
• cert: Raise an error for broken certificate file
• cert: Set TLS ciphers list correcty on older OpenSSL, e.g. openssl v1.0.2

Enjoy logging!

Read More

Fluentd v1.11.1 has been released

Hi users!

We have released v1.11.1. ChangeLog is here.

in_http: Improve time field handling

in_http now supports time parsing in record field for default json/msgpack request. Here is configuration example:

<source>
  @type http
  @id input_http
  port 8888
  <parse>
    time_format %iso8601
    time_key logtime
    keep_time_key true
  </parse>
</source>

<match test.**>
  @type stdout
</match>

If you post following requests:

# ruby code example

require 'net/http'
require 'json'

record  = {'json' => {'k' => 'hello1', 'logtime' => '2020-06-10T11:14:27+09:00'}.to_json}
records = {'json' => [
  {'k' => 'hello2', 'logtime' => '2020-06-10T11:14:28+09:00'},
  {'k' => 'hello3', 'logtime' => '2020-06-10T11:14:29+09:00'}
].to_json}

def post(path, params)
  http = Net::HTTP.new('127.0.0.1', 8888)
  req  = Net::HTTP::Post.new(path, {})
  req.set_form_data(params)
  http.request(req)
end

post("/test.http", record)
post("/test.http", records)

the log shows expected result:

2020-06-10 11:14:27.000000000 +0900 test.http: { "k":"hello1","logtime":"2020-06-10T11:14:27+09:00"}
2020-06-10 11:14:28.000000000 +0900 test.http: { "k":"hello2","logtime":"2020-06-10T11:14:28+09:00"}
2020-06-10 11:14:29.000000000 +0900 test.http: { "k":"hello3","logtime":"2020-06-10T11:14:29+09:00"}

Before, this configuration doesn't work because in_http assumes the time value is floating point.

Major bug fixes

• in_tail: Use actual path instead of base pattern for ignore list
• child_process helper: Fix child process failure due to SIGPIPE if the command uses stdout

Enjoy logging!

Read More

Fluentd v1.11.0 has been released

Hi users!

We have released v1.11.0. ChangeLog is here.

in_unix: Use v1 API

Migrate to v1 plugin API.

In addition, in_unix now supports tag parameter to use fixed tag.

Improve syslog parser

There are several improvements:

• Improve message_format auto performance by avoiding object allocation
• Support any time_format for RFC3164 with parser_type string
• Support parser_type string for RFC5424. This parser is 2x faster than parser_type regexp

We now recommend to use parser_type string for both RFC3164 and RFC5424.

Major bug fixes

• in_gc_stat: Add use_symbol_keys parameter to support non-stdout plugin.

Enjoy logging!

Read More

Fluentd v1.10.4 has been released

Hi users!

We have released v1.10.4. ChangeLog is here. This release fixes several bugs.

out_http: Support single json array payload

out_http uses ndjson for json payload by default.

{"key":"value1"}
{"key":"value2"}
...
{"key":"valueN"}

You can change payload format to single json array by setting json_array true.

# conf
<match pattern>
  @type http
  # other parameters...
  json_array true
  <format>
    @type json
    add_newline false
  </format>
</match>

# payload
[{"key":"value1"},{"key":"value2"},...,{"key":"valueN"}]

Major bug fixes

• supervisor: Call File.umask(0) for standalone worker
• out_forward: Fix ZeroDivisionError issue with weight 0

Enjoy logging!

Read More

Fluentd v1.10.3 has been released

Hi users!

We have released v1.10.3. ChangeLog is here. This release fixes several bugs.

record accessor helper: Support set method

This feature is for record mutation in the plugin. Here is code example:

record = {'key1' => {'key2' => "value"}}

accessor = record_accessor_create("$['key1']['key2']")
accessor.set(r, "new value!") # record is {'key1' => {'key2' => "new value!"}} 

config: Ruby DSL format is now deprecated

We decided to deprecate experimental Ruby DSL configuration by following reasons:

• Hard to maintain full configuation features
• Hard to implement/parse in other system

Major bug fixes

• outforward: Disable `lingertimeout` setting on Windows
• service discovery: Fix warning of service discovery manager when fluentd stops

Enjoy logging!

Read More