Fluentd Blog

Fluentd v1.6.1 and v1.6.2 has been released

Hi users!

We have released v1.6.1 and v1.6.2. ChangeLog is here. These release includes several enhancements and bug fixes.

http_server helper: Add title argument to the API

This fix is important to launch HTTP servers in different plugins. Old implementation uses same thread identifier for all HTTP servers and it won't work.

Check also API article.

certificates: Support all private keys OpenSSL supports

fluentd assumes private key is RSA before. Now, other types are supported, e.g. EC and DSA.

Other bug fixes

• pluginconfigformatter: use new doc URL
• out_forward: Avoid zero division error when there are no available nodese

Enjoy logging!

Read More

Fluentd v1.6.0 has been released

Hi users!

We have released v1.6.0. ChangeLog is here. This release includes new plugin helper and some improvement.

http_server plugin helper

This plugin helper is for in_monitor_agent like plugins. Here is code example:

module Fluent::Plugin
  class FooServerInput < Input
    Fluent::Plugin.register_input('foo_server', self)
    helpers :http_server

    # ...

    def start
      super

      create_http_server(addr: @bind, port: @port, logger: log) do |serv|
        serv.get('/api/hello') { |req|
          # Return value is an Array which consists of status code, headers and body
          [200, {'Content-Type' => 'text/plain'}, 'hello get']
        }
        serv.post('/api/hello') { |req|
          [200, {'Content-Type' => 'text/plain'}, 'hello post']
        }
        # support other HTTP method
      end
    end
  end
end

This helper uses WEBrick by default. But if you install async-http gem, use async-http instead for the performance. in_monitor_agent replaced HTTP server implemenation with http_server helper.

in_monitor_agent: Add more buffer/output metrics

Since v1.6.0, monitor_agent input plugin exposes more buffer/output metrics. Here is the comparison.

# v1.6
{
  "plugin_id": "out_forward",
  "plugin_category": "output",
  "type": "forward",
  "output_plugin": true,
  "buffer_queue_length": 0,
  "buffer_timekeys": [],
  "buffer_total_queued_size": 0,
  "retry_count": 0,
  "emit_records": 0,
  "emit_count": 0,
  "write_count": 0,
  "rollback_count": 0,
  "slow_flush_count": 0,
  "flush_time_count": 0,
  "buffer_stage_length": 0,
  "buffer_stage_byte_size": 0,
  "buffer_queue_byte_size": 0,
  "buffer_available_buffer_space_ratios": 100,
  "retry": {}
}

# v1.5 or earlier
{
  "plugin_id": "out_forward",
  "plugin_category": "output",
  "type": "forward",
  "output_plugin": true,
  "buffer_queue_length": 0,
  "buffer_timekeys": [],
  "buffer_total_queued_size": 0,
  "retry_count": 0,
  "retry": {}
}

New fields are below:

• emit_records: The total number of emitted records
• emit_count: The total number of emit call in output plugin
• write_count: The total number of write/try_write call in output plugin
• rollback_count: The total number of rollback. rollback happens when write/try_write failed
• slow_flush_count: The total number of slow flush. This count will be incremented when buffer flush is longer than slow_flush_log_threshold
• flush_time_count: The total time of buffer flush in milliseconds
• buffer_stage_length: Current lenght of staged buffer chunks
• buffer_stage_byte_size: Current bytesize of staged buffer chunks
• buffer_queue_byte_size: Current bytesize of queued buffer chunks
• buffer_available_buffer_space_ratios: Show available space for buffer

socket/out_forward: Add connect_timeout parameter for TCP/TLS

This socket timeout is used for connection establishment. If your DNS returns wrong IP or machine, connection establishment waits a long time, e.g. 180 seconds by system default. connect_timeout parameter avoids this kind of problem.

Major bug fixes

• cert: Fix cert pattern to accept certificates which don't end with newline

Enjoy logging!

Read More

Fluentd v1.5.2 has been released

Hi users!

We have released v1.5.2. ChangeLog is here. This release is for fixing out_forward keepalive issue. If you want to use keepalive and <security>, update fluentd to v1.5.2.

Enjoy logging!

Read More

Fluentd v1.5.1 has been released

Hi users!

We have released v1.5.1. ChangeLog is here. This release includes several enhancements and bug fixes.

in_tail: Increase read block size to reduce IO call

We changed read block size from 2048 to 8192. 2048 is small for large logs. in_tail uses IO#readpartial method and the overhead of this method is bit higher. So decreasing the number of method call reduces CPU usage.

Major bug fixes

• out_forward: Fix the socket handling of keepalive
• parser: Fix the use of name based timezone
• in_monitor_agent: Fix debug parameter handling

Enjoy logging!

Read More

Fluentd v1.5.0 has been released

Hi users!

We have released v1.5.0. ChangeLog is here. This release includes nice features and improvements.

out_forward: Support keepalive feature

<source>
  @type tail
  # ... other parameters
</source>

<match pattern>
  @type forward
  # ... other parameters
  keepalive true
  keepalive_timeout 60s # optional. Default is no timeout
</match>

forward output plugin now supports keepalive/keepalive_timeout parameters. This feature is useful to reduce the connection cost per flush. In addition, this feature is also good for load balancer situation.

in_http/in_syslog: Support TLS transport

These plugins now support <transport tls> configuration. Here is syslog input / rsyslog config example for TLS:

# fluentd
<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  <transport tls>
    ca_path /etc/pki/ca.pem
    cert_path /etc/pki/cert.pem
    private_key_path /etc/pki/key.pem
    private_key_passphrase PASSPHRASE
  </transport>
  tag system
</source>

# rsyslog
$DefaultNetstreamDriverCAFile /etc/pki/ca.pem
$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
*.* @@127.0.0.1:5140

http input supports same parameters with <transport tls>.

in_forward: Add tag/add_tag_prefix parameters

You can change incoming event tag in forward input without additional plugins.

<source>
  @type forward
  port 24230
  tag system1.log # change tag to system1.log
</source>

<source>
  @type forward
  port 24240
  add_tag_prefix system2 # add prefix to incoming tag, e.g. system2.app.log
</source>

For event routing, we can use label feature but it can't modify event tag. These parameters are useful when you want to organize incoming tag in the aggregator.

Major bug fixes

• out_forward: Don't use SO_LINGER on SSL/TLS WinSock.
• server: Fix recursive lock issue in TLSServer

Enjoy logging!

Read More