Fluentd Blog

Fluentd v1.11.5 has been released

Hi users!

We have released v1.11.5. ChangeLog is here.

formatter: Provide newline parameter to support CRLF

This parameter is for Windows environment. Users can choose LF or CRLF newline for add_newline true case.

newline lf # or crlf

out_http: adding support for intermediate certificates

Add support for sending a client certificate chain for mutual TLS authentication in the HTTP output plugin.

Major bug fixes

• Fix a bug that windows service isn't stopped gracefuly

Enjoy logging!

Read More

Fluentd Ecosystem Survey 2020

Hey All,

Thank you for being part of the Fluent Slack Community! As part of improving Fluentd and Fluent Bit, we want to know what pains and feedback you are experiencin to better manage our future together.

We have created an anonymous survey that allows you to share this here.

Optionally, if you are interested you can add your email to give us direct feedback. Looking forward to your feedbacks!

Read More

Fluentd v1.11.3 has been released

Hi users!

We have released v1.11.3. ChangeLog is here.

log: Add ignore_same_log_interval parameter

Fluentd has ignore_repeated_log_interval parameter but this is not enough for some frequent log generation case. For example, if the plugin generates several log messages in one action, logs are not repeated:

# retry generates same error messages with ignore_repeated_log_interval
def write(chunk)
  # process 1...
  log.error "message1"
  # process 2...
  log.error "message2"
end

ignore_same_log_interval resolves these cases. With this option, logger keeps to store all log messages for validation.

<system>
  ignore_same_log_interval 60s
</system>

inexec: Add `connectmode` parameter to read stderr

in_exec doesn't read stderr and this is not good for script debug. To resolve this problem, add connect_mode parameter to read stderr logs.

<source>
  @type exec
  tag somescript
  command python /path/to/script.py
  connect_mode read_with_stderr
  <parse>
    @type none
  </parse>
</source>

Major bug fixes

• buffer: Fix calculation of timekey stats
• buffer: fix binmode usage for prevent gc

Enjoy logging!

Read More

Fluentd v1.11.2 has been released

Hi users!

We have released v1.11.2. ChangeLog is here.

Allow regular expression in filter/match tag matching

Regular expression, /pattern/, is now allowed in tag matching:

<source>
  @type forward
</source>

# For a.xxx tag
<filter a.**>
  @type record_transformer
  <record>
    new_key a-started tag record
  </record>
</filter>

# For non a.xxx tag like b.xxx/c.xxx/...
<filter /(?!a\.).*/>
  @type record_transformer
  <record>
    new_key other tag record
  </record>
</filter>

<match **>
  @type stdout
</match>

If you send following events:

% echo '{"msg":"hi"}' | fluent-cat a.foo
% echo '{"msg":"hi"}' | fluent-cat b.foo

the log shows:

2020-08-06 13:40:03.111196000 +0900 a.foo: {"msg":"hi","new_key":"a-started tag record"}
2020-08-06 13:40:09.492380000 +0900 b.foo: {"msg":"hi","new_key":"other tag record"}

Of course, simple data pipeline is important for robust log collection. If you need this feature, check your configuration/data flow first.

in_dummy is renamed in_sample

We recommend to use sample instead of dummy:

<source>
  @type sample
  sample {"hello":"world"}
</source>

# If you use fluentd v1.11.1 or earlier, use following configuration
# Fluentd v2 will remove old configuration support
<source>
  @type dummy
  dummy {"hello":"world"}
</source>

Major bug fixes

• buffer: Fix log message for chunk_limit_records case
• buffer: Fix timekey optimization for non-windows platform
• cert: Raise an error for broken certificate file
• cert: Set TLS ciphers list correcty on older OpenSSL, e.g. openssl v1.0.2

Enjoy logging!

Read More

Fluentd v1.11.1 has been released

Hi users!

We have released v1.11.1. ChangeLog is here.

in_http: Improve time field handling

in_http now supports time parsing in record field for default json/msgpack request. Here is configuration example:

<source>
  @type http
  @id input_http
  port 8888
  <parse>
    time_format %iso8601
    time_key logtime
    keep_time_key true
  </parse>
</source>

<match test.**>
  @type stdout
</match>

If you post following requests:

# ruby code example

require 'net/http'
require 'json'

record  = {'json' => {'k' => 'hello1', 'logtime' => '2020-06-10T11:14:27+09:00'}.to_json}
records = {'json' => [
  {'k' => 'hello2', 'logtime' => '2020-06-10T11:14:28+09:00'},
  {'k' => 'hello3', 'logtime' => '2020-06-10T11:14:29+09:00'}
].to_json}

def post(path, params)
  http = Net::HTTP.new('127.0.0.1', 8888)
  req  = Net::HTTP::Post.new(path, {})
  req.set_form_data(params)
  http.request(req)
end

post("/test.http", record)
post("/test.http", records)

the log shows expected result:

2020-06-10 11:14:27.000000000 +0900 test.http: { "k":"hello1","logtime":"2020-06-10T11:14:27+09:00"}
2020-06-10 11:14:28.000000000 +0900 test.http: { "k":"hello2","logtime":"2020-06-10T11:14:28+09:00"}
2020-06-10 11:14:29.000000000 +0900 test.http: { "k":"hello3","logtime":"2020-06-10T11:14:29+09:00"}

Before, this configuration doesn't work because in_http assumes the time value is floating point.

Major bug fixes

• in_tail: Use actual path instead of base pattern for ignore list
• child_process helper: Fix child process failure due to SIGPIPE if the command uses stdout

Enjoy logging!

Read More