Fluentd Blog

Fluentd v1.19.0 has been released

Hi users!

We have released v1.19.0 on 2025-07-30. ChangeLog is here.

This release is a new release of v1.19 series. This release includes many new features and bug fixes, with a particular focus on reducing operational and running costs and improving overall stability.

$ rm Gemfile.lock
$ bundle

Major new features and improvements

Add feature to evacuate chunk files when retry limit

In this release, we added a buffer evacuation feature to the buf_file and buf_file_single plugins. This simplifies recovery after failure.

In previous versions, when an output plugin fails to deliver data by network issue, if the retry limit was exceeded and secondary output was not configured, the data would be discarded.

In this release, buffer files are automatically evacuated when the retry limit is exceeded. Once the issue is resolved, you can restore the evacuated files to the original buffer directory and restart Fluentd to resume sending the data.

You can even restart without causing downtime to input plugins like in_udp by using the zero-downtime restart feature.

Evacuated files are stored at:

${root_dir}/buffer/${plugin_id}/

Ref. Handling Successive Failures

Note: This feature is disabled when secondary outputs or retry_forever are used. To enable buffer evacuation, remove those settings.

Improved buffer corruption detection

Buffer files may become corrupted due to events like system crashes or sudden power down. In previous versions, Fluentd could fail to handle corrupted files properly, triggering errors on every restart. You had to manually identify and delete the corrupted files.

This release improves buffer corruption detection. When corruption is detected, the file is automatically moved to a backup directory, and startup errors are suppressed.

Enhanced metrics

We have improved Fluentd's observability by enhancing its metrics capabilities:

• Input plugin metrics are now enabled by default.
  • Previously, you needed to explicitly set enable_input_metrics to true. From this release, it's collected by default. To disable, set enable_input_metrics false.
• New metrics for in_tail:
  • tracked_file_count: Number of files currently being tailed.
• New metrics for output plugin:
  • write_secondary_count: Cumulative count of records written to secondary output.
  • drop_oldest_chunk_count: Cumulative count of chunks dropped when drop_oldest_chunk is used for overflow_action.

Performance improvement: switch from yajl-ruby to json gem

Previously, we used yajl-ruby for high-performance JSON processing, such as writing buffer chunks and parsing with parser_json.

However, in recent years, the standard Ruby json library has improved performance significantly. So we changed to use the json gem as the default JSON processor.

Zstandard (zstd) compression support

We have added support for the Zstandard (zstd) compression format, which offers both high compression ratios and faster processing compared to gzip.

• The following plugins now accept compress zstd:
  • buffer
  • out_file
  • out_forward

Note: To use zstd with out_forward, the destination server should support it. You cannot use it with Fluent Bit or Fluentd versions older than v1.19.0.

TLS 1.3 support

The out_http plugin now supports TLS 1.3, offering enhanced security. By default, TLS 1.2 is used. To enable TLS 1.3, set the tls_version parameter as follows:

<match tag>
  @type http
  ...
  tls_version TLSv1_3
  ...
</match>

Other useful improvements

• in_forward plugin:
  • skip_invalid_event is now enabled by default to skip corrupted records.
• out_stdout plugin:
  • Added the use_logger parameter. Setting use_logger false forces output to STDOUT regardless of Fluentd's own logging configuration. This is especially useful in container environments like Kubernetes.
• out_file plugin:
  • Added symlink_path_use_relative to allow relative paths in symlink_path.
• System configuration:
  • Added forced_stacktrace_level to control the log level of stack traces.

Major bug fixes

Fixed memory leak in formatter_csv plugin

To improve performance, it introduced a caching mechanism in formatter_csv, but incorrectly cached threads. This caused memory leaks when used with in_exec, which spawns a new thread per log entry.

This fix is backported to Fluentd v1.16.8.

in_tail plugin: fixed incorrect behavior with encoding option

The in_tail plugin provides encoding and from_encoding options.

• Expected usage:
  • Setting only encoding: Specifying encoding information.
  • Setting both encoding and from_encoding: Converting encoding.

Due to a bug since Fluentd v0.14.12 (at least from td-agent v3), when only encoding was set, unintended encoding conversion occurred, it might corrupt log data.

In previous versions, you need to specify the same encoding for both encoding and from_encoding as workaround.

Windows: fixed bug where service doesn't stop when the supervisor is dead

On Windows, if the supervisor process exited due to configuration errors, the service could continue running incorrectly. This has been fixed so the service now stops as expected, improving error detection.

This fix is backported to Fluentd v1.16.8.

Fixed --umask Bug

Fixed a bug where the --umask setting was ignored when used together with --daemon, leading to unexpected file permissions.

Enjoy logging!