Fluentd v1.1.1 has been released

Hi users!

We have released v1.1.1. ChangeLog is here. This release includes several enhancements and bug fixes..

in_forward/server helper: Support mutual TLS authentication

We added ca_path and client_cert_path parameters to server plugin helper. You can now send data between fluent-bit and fluentd with mutual TLS authentication. In in_forward, put these parameters into <transport tls>.

  @type forward
  # ... other parameters
  <transport tls>
    # ... other parameters
    ca_path /path/to/ca_file
    client_cert_auth true

buf_file: Skip and delete broken chunk files during resume

fluentd can't restart if unexpected broken file exist in buffer directory, because such files cause errors in resume routine. Since v1.1.1, if fluentd found broken chunks during resume, these files are skipped and deleted from buffer directory.

We have a plan to add backup feature for broken files in v1.2.0. It moves broken files into backup directory instead of deletion.

Major bug fixes

  • json parser: Fix error handling for oj 3.x
  • in_tail: Fix watcher race condition during shutdown
  • in_http: Emit event time instead of raw time value in batch mode

Thanks for submitting bug reports and patches :)

Enjoy logging!

Read More

Fluentd v1.1.0 has been released

Hi users!

We have released v1.1.0. ChangeLog is here. This release includes several new features.

config: Support hostname and worker_id short-cut in embed Ruby code

Fluentd configuration supports embedded ruby code in "#{}" string. Many users use this feature to embed runtime value in plugin parameters. This is conf example:

@id "out_foo#{ENV['SERVERENGINE_WORKER_ID']}}" # add worker id to plugin id under multi-process environment
tag "log.#{Socket.gethostname}"                # Use hostname in tag parts
  metadata "#{ENV['SERVER_ROLE']}-#{ENV['FOO']}"

We noticed setting hostname and worker_id is popular and current configuration is bit messy because it depends on fluentd internal, e.g. SERVERENGINE_WORKER_ID comes from serverengine used in Fluentd. So we added hostname and worker_id short-cut to cover popular cases. Here is new conf:

@id "out_foo#{worker_id}" # add worker id to plugin id under multi-process environment
tag "log.#{hostname}"     # Use hostname in tag parts

If other popular case found, we will add new short-cut.

record_accessor helper: Support nested field deletion

We have record_accessor helper for accessing nested field. Since v1.1.0, this helper supports nested field deletion. This feature is useful in record_transformer like plugins.

Syntax is same and you can delete nested field via accessor object in your plugin code.

deleter = record_accessor_create("$.key1.key2")
deleter.delete(record) # delete record["key1"]["key2"] field

record_transfomer filter plugin supports this feature with remove_keys parameter.

fluent-ca-generate: command for certificates generation

This is port from fluent-plugin-secure-forward. Use-case is same with Using private CA file and key. Change command name to fluent-ca-generate.

You can change several values like CN/country/etc via command option. Check --help for all options.

Major bug fixes

  • server helper: Fix TCP + TLS degradation

We changes buffer management in in_tcp plugin and it breaks TLS socket handling. We changed TLS socket structure and fixed the regression.

Enjoy logging!

Read More

Fluentd v1.0.2 has been released

Hi users!

We have released v1.0.2. ChangeLog is here. This announcement includes v1.0.1 changes.

in_udp: Add receive_buffer_size parameter

This parameter is corresponding to SO_RCVBUF of socket. If you want to improve the performance of in_udp, set larger size to this parameter.

Major bug fixes

  • in_tail: Handle truncated/moved back case

In rename(file -> file.bak) -> truncate -> rename(file.bak -> file) case, fluentd reads logs from the bottom. It causes log lost when application appends logs after truncate. Latest version reads logs from the head.

  • out_forward: Rebuild weight array to apply server setting on active/standby environment

After active node is backed, standby node was still listed in active nodes. v1.0.2 fixes this regression. standby node is excluded from active nodes properly.

We have released v0.12.42 together. v0.12.42 includes same bug fixes.

Enjoy logging!

Read More

Fluentd v1.0

We announced Fluentd v1.0 at CloudNativeCon + KubeCon NA 2017.

See CNCF blog about detailed information: Fluentd v1.0 - Cloud Native Computing Foundation

The important point is v1.0 is built on top of v0.14 stable version. No need changes for upgrading from v0.14 to v1.0.

If you are interested in v1.0 features, see following slide:

Fluentd v1.0 in a nutshell from N Masahiro

Fluentd v0.12 is maintenance phase

We continue to update fluentd v0.12 but the main changes are backport and security fix. We focus on v1.0 development.

Docker image changes

We have a plan to change stable tags used version from v0.12 to v1.0 at Jan 1, 2018. If you want to keep to use v0.12 serise in your environment, specify v0.12 tag instead of stable/latest tags.

In addition, we don't update v0.14 tags anymore. Use v1.0 tags instead.

Finally, thanks all! We start new journey with you :)

Read More

Fluentd v1.0.0.rc1/v0.14.25 has been released

Hi users!

We have released v1.0.0.rc1 for preparing stable v1.0 release. ChangeLog is here.

We have released v0.14 stable before and it means new important features and plugin APIs are fixed. The purpose of v0.14 develops new features for v1.0 so time to release v1.0.

v1.0 announcement is good for fluentd development and users. We can separate new feature / bug fix release by Semantic versioning. It makes fluentd version management easier than before. The important point is no differences between v1.0 and v0.14 :)

We are now fixing reported bugs after v0.14 stable release. We will release v1.0.0 soon :)

Major bug fixes

  • engine: Fix plugin shutdown sequence

We introduced new plugin lifecycle control but it causes flush timing mismatch between input and output. This bug is fixed by combine before_shutdown and shutdown sequence in one call.

  • supervisor: Fix --user/--group parameter regression

We have called change privilege routine twice since v0.14.16 and it causes permission error at startup. If you have this error with v0.14.23 or earlier, update version fixed the error.

We have released v0.14.25 together. This is same as v1.0.0.rc1. If your used plugin depends on only fluentd v0.14, use this version temporarily.

For plugin developers:

v1.0 and v0.14 plugin API are same so v1.0 works with existing plugins. If the plugin uses add_runtime_dependency "fluentd", "~> 0.14.0" like setting in the gemspec, update it to add_runtime_dependency "fluentd", [">= 0.14.0", "< 2"].

Enjoy logging!

Read More

About Fluentd

Fluentd is an open source data collector to simplify log management.


Want to learn more about Fluentd? Check out these pages.

Follow Us!