Fluentd v1.1.0 has been released

Hi users!

We have released v1.1.0. ChangeLog is here. This release includes several new features.

config: Support hostname and worker_id short-cut in embed Ruby code

Fluentd configuration supports embedded ruby code in "#{}" string. Many users use this feature to embed runtime value in plugin parameters. This is conf example:

@id "out_foo#{ENV['SERVERENGINE_WORKER_ID']}}" # add worker id to plugin id under multi-process environment
tag "log.#{Socket.gethostname}"                # Use hostname in tag parts
<record>
  metadata "#{ENV['SERVER_ROLE']}-#{ENV['FOO']}"
</record>

We noticed setting hostname and worker_id is popular and current configuration is bit messy because it depends on fluentd internal, e.g. SERVERENGINE_WORKER_ID comes from serverengine used in Fluentd. So we added hostname and worker_id short-cut to cover popular cases. Here is new conf:

@id "out_foo#{worker_id}" # add worker id to plugin id under multi-process environment
tag "log.#{hostname}"     # Use hostname in tag parts

If other popular case found, we will add new short-cut.

record_accessor helper: Support nested field deletion

We have record_accessor helper for accessing nested field. Since v1.1.0, this helper supports nested field deletion. This feature is useful in record_transformer like plugins.

Syntax is same and you can delete nested field via accessor object in your plugin code.

deleter = record_accessor_create("$.key1.key2")
deleter.delete(record) # delete record["key1"]["key2"] field

record_transfomer filter plugin supports this feature with remove_keys parameter.

fluent-ca-generate: command for certificates generation

This is port from fluent-plugin-secure-forward. Use-case is same with Using private CA file and key. Change command name to fluent-ca-generate.

You can change several values like CN/country/etc via command option. Check --help for all options.

Major bug fixes

  • server helper: Fix TCP + TLS degradation

We changes buffer management in in_tcp plugin and it breaks TLS socket handling. We changed TLS socket structure and fixed the regression.

Enjoy logging!

Read More

Fluentd v1.0.2 has been released

Hi users!

We have released v1.0.2. ChangeLog is here. This announcement includes v1.0.1 changes.

in_udp: Add receive_buffer_size parameter

This parameter is corresponding to SO_RCVBUF of socket. If you want to improve the performance of in_udp, set larger size to this parameter.

Major bug fixes

  • in_tail: Handle truncated/moved back case

In rename(file -> file.bak) -> truncate -> rename(file.bak -> file) case, fluentd reads logs from the bottom. It causes log lost when application appends logs after truncate. Latest version reads logs from the head.

  • out_forward: Rebuild weight array to apply server setting on active/standby environment

After active node is backed, standby node was still listed in active nodes. v1.0.2 fixes this regression. standby node is excluded from active nodes properly.

We have released v0.12.42 together. v0.12.42 includes same bug fixes.

Enjoy logging!

Read More

Fluentd v1.0

We announced Fluentd v1.0 at CloudNativeCon + KubeCon NA 2017.

See CNCF blog about detailed information: Fluentd v1.0 - Cloud Native Computing Foundation

The important point is v1.0 is built on top of v0.14 stable version. No need changes for upgrading from v0.14 to v1.0.

If you are interested in v1.0 features, see following slide:

Fluentd v1.0 in a nutshell from N Masahiro

Fluentd v0.12 is maintenance phase

We continue to update fluentd v0.12 but the main changes are backport and security fix. We focus on v1.0 development.

Docker image changes

We have a plan to change stable tags used version from v0.12 to v1.0 at Jan 1, 2018. If you want to keep to use v0.12 serise in your environment, specify v0.12 tag instead of stable/latest tags.

In addition, we don't update v0.14 tags anymore. Use v1.0 tags instead.

Finally, thanks all! We start new journey with you :)

Read More

Fluentd v1.0.0.rc1/v0.14.25 has been released

Hi users!

We have released v1.0.0.rc1 for preparing stable v1.0 release. ChangeLog is here.

We have released v0.14 stable before and it means new important features and plugin APIs are fixed. The purpose of v0.14 develops new features for v1.0 so time to release v1.0.

v1.0 announcement is good for fluentd development and users. We can separate new feature / bug fix release by Semantic versioning. It makes fluentd version management easier than before. The important point is no differences between v1.0 and v0.14 :)

We are now fixing reported bugs after v0.14 stable release. We will release v1.0.0 soon :)

Major bug fixes

  • engine: Fix plugin shutdown sequence

We introduced new plugin lifecycle control but it causes flush timing mismatch between input and output. This bug is fixed by combine before_shutdown and shutdown sequence in one call.

  • supervisor: Fix --user/--group parameter regression

We have called change privilege routine twice since v0.14.16 and it causes permission error at startup. If you have this error with v0.14.23 or earlier, update version fixed the error.

We have released v0.14.25 together. This is same as v1.0.0.rc1. If your used plugin depends on only fluentd v0.14, use this version temporarily.

For plugin developers:

v1.0 and v0.14 plugin API are same so v1.0 works with existing plugins. If the plugin uses add_runtime_dependency "fluentd", "~> 0.14.0" like setting in the gemspec, update it to add_runtime_dependency "fluentd", [">= 0.14.0", "< 2"].

Enjoy logging!

Read More

Fluentd v0.14.23 has been released

Hi users!

We have released v0.14.23. This release is mainly for bug fix.

Here are the changes of v0.14.23(full ChangeLog is available here)

in_udp: Add remove_newline parameter

Current in_udp removes the newline from the end of incoming payload. This is for one line log processing. But originally, UDP is message-oriented and no need such processing. In addition, this behaviour is not good for binary message.

So we add remove_newline parameter to control this behaviour. Default is true for the compatibility, but we will change the default value to false after 6-months transition period.

Major bug fixes

  • in_tcp: Fix log corruption under load.

in_tcp shared the buffer between connections and it has a log corruption problem under high load. New implementation has a buffer inside the connection so the problem is fixed.
Patch from @AM-iain. Thanks!

  • out_forward: Fix elapsed time miscalculation in tcp heartbeat

TCP heartbeat misjudges the server failure when one of server in <server>s has a problem. It causes detaching active destination from target server list. We fixed this problem by changing elapsed time calculation routine.
UDP heartbeat doesn't have this problem.

We will release v0.12.41 with several backports soon.

Enjoy logging!

Read More

About Fluentd

Fluentd is an open source data collector to simplify log management.

Learn

Want to learn more about Fluentd? Check out these pages.

Follow Us!