Fluentd Blog

Fluentd v1.4.1 has been released

Hi users!

We have released v1.4.1. ChangeLog is here. This release includes several enhancements and bug fixes.

system: Add worker_id to process_name when workers is larger than 1

This improves the multi-workers operation.

<system>
  process_name fluentd_aggr
  workers 2
</system>

With this configuration, older version shows same process name for workers.

# ps result
repeatedly       94982   0.0  0.2  2521484  29948 s004  S+    9:56PM   0:00.48 worker:fluentd_aggr  
repeatedly       94983   0.0  0.2  2522508  28304 s004  S+    9:56PM   0:00.49 worker:fluentd_aggr

Since v1.4.1, process name has worker id to identify the process.

# ps result
repeatedly       94879   0.0  0.2  2522508  28808 s004  S+    9:55PM   0:00.49 worker:fluentd_aggr0
repeatedly       94880   0.0  0.2  2521484  29084 s004  S+    9:55PM   0:00.49 worker:fluentd_aggr1

Major bug fixes

• out_forward: Make tls_client_private_key_passphrase secret
• in_syslog: Check message length when read from buffer in octet counting

Enjoy logging!

Read More

Fluentd v1.4.0 has been released

Hi users!

We have released v1.4.0. ChangeLog is here. This release includes nice features and improvements.

multiprocess: Support <worker N-M> syntax

This feature enables grouping workers. Here is configuration example:

<system>
  workers 5
</system>

<worker 0-1>
  <source>
    @type forward
  </source>

  <match test.**>
    @type stdout
  </match>
</worker>

<worker 2-4>
  <source>
    @type tcp
    <parse>
      @type json
    </parse>
    tag test
  </source>

  <match test>
    @type stdout
  </match>
</worker>

With this configuration, worker 0/1 launches forward input with 24224 port and worker 2/3/4 launches tcp input with 5170 port.

out_file: Support placeholders in symlink_path parameters

This change improves symlink_path usecase. You can tail multiple files based on placeholders. Here is a configuration and result example:

# config
<source>
  @type dummy
  tag dummy1
</source>

<source>
  @type dummy
  tag dummy2
</source>

<match dummy*>
  @type file
  path /tmp/logs/${tag}
  symlink_path /tmp/logs/current-${tag}
  <buffer tag,time>
    @type file
  </buffer>
</match>

# result
$ tree /tmp/logs/
/tmp/logs/
├── ${tag}
│   ├── buffer.b57fb1dd96306dd0b308e094f7ec2228f.log
│   ├── buffer.b57fb1dd96306dd0b308e094f7ec2228f.log.meta
│   ├── buffer.b57fb1dd96339a870530991d4871cfe11.log
│   └── buffer.b57fb1dd96339a870530991d4871cfe11.log.meta
├── current-dummy1 -> /tmp/logs/${tag}/buffer.b57fb1dd96339a870530991d4871cfe11.log
└── current-dummy2 -> /tmp/logs/${tag}/buffer.b57fb1dd96306dd0b308e094f7ec2228f.log

config: Support embedded ruby code in section argument.

You can use "#{ruby code}" syntax in <filter>/<match> argument. Let's see configuration example:

<match "app.#{ENV['FLUENTD_TAG']}">
  @type stdout
</match>

If you set dev to FLUENTD_TAG environment variable, this config becomes <match app.dev>.

Major bug fixes

• server: Ignore EPIPE in TLS accept to avoid fluentd restart
• server: Ignore IOError and unrecoverable errors in UDP

Enjoy logging!

Read More

Fluentd v1.3.3 has been released

Hi users!

We have released v1.3.3. ChangeLog is here. This release is mainly for bug fixes

parser_syslog: Support RFC5424 timestamp without subseconds

Now, parser_syslog can accept RFC5424 timestampe with/without subseconds.

<16>1 2017-02-06T13:14:15Z 192.168.0.1 fluentd - - - Hi, from Fluentd!
<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi, from Fluentd with subseconds!

No additional setting for these events.

Major bug fixes

• server: Ignore ECONNRESET in TLS accept to avoid fluentd restart
• log: Fix plugin logger ignores fluentd log event setting This fixes @id having plugins ignore <label @FLUENT_LOG> configuration.

Enjoy logging!

Read More

Fluentd v1.3.0 has been released

Hi users!

We have released v1.3.0. ChangeLog is here.

Output: Change thread execution control

This is internal change of output plugin. This change should fix unexpected error like zlib's DataError.

Output: Change default value of queued_chunks_limit_size

queued_chunks_limit_size is now same as flush_thread_count when queued_chunks_limit_size is not specified.

Before, queued_chunks_limit_size is unlimited. This causes critical situation with smaller flush_interval when the destinaion has a problem. For example, lots of very small queued buffer are created and it causes resource limit error.

Unlimited queued buffer is unexpected behaviour, so we changed this behaviour for safety.

in_syslog: octet counting support

We introduced frame_type parameter. With frame_type octet_count configuration, in_syslog accepts octet counting messages like below.

44 <6>Sep 10 00:00:00 localhost logger: hello!

Major bug fixes

• server: Fix deadlock between on_writable and close in sockets. This fixes fluentd restart issue with in_forward's TLS setting error.

Other improvements

• in_http: Add support for CORS preflight requests. Now you can send cross-domain AJAX requests with JSON payload to Fluentd.

Enjoy logging!

Read More

Fluentd v1.2.5 has been released

Hi users!

We have released v1.2.5. ChangeLog is here. This release fixed in_tail resource leak.

Major bug fixes

• in_tail: Fix log rotation related resource leak. If you use in_tail with log rotation files, we recommend to update fluentd to this version.

Enjoy logging!

Read More