Fluentd v1.13.0 has been released

Hi users!

We have released v1.13.0. ChangeLog is here.

There are some topics in this release.

  • service_discovery: extended to support service discovery manager in simpler way
  • in_tail: handling log throttling per file feature has been supported
  • in_http: HTTP GET requests has been supported
  • The log rotate settings in system configuration has been supported
  • fluent-cat: the issue resending secondary file in specific format has been fixed

There are many in_tail bug fixes, we recommend to upgrade to v1.13.0.

service_discovery: extended to support service discovery manager in simpler way

In the previous versions, if plugin author supports service_discovery, it must build configurations structure explicitly, but it is complicated way.

Since Fluentd v1.13.0, it is enough to call #service_discovery_configure from Plugin#configure. It makes implementation costs lower.

in_tail: handling log throttling per file feature has been supported

In the previous versions, there is no way to limit log throttling.

Since Fluentd v1.13.0, the configuration parameter - read_bytes_limit_per_second is introduced.

<source>
  @type tail
  ...
  read_bytes_limit_per_second 4k
</source>

As read_bytes_limit_per_second accepts numbers with SI prefix such as 4k.

in_http: HTTP GET requests has been supported

This functionality is implemented to support health check feature on Azure App.

Azure App uses GET requests to check if the HTTP server is working all right. In the previous versions of Fluentd, it responds with "400 Bad Requests" to HTTP GET, so it does not work well for health checking purpose.

Since Fluentd v1.13.0, Fluentd returns a "200 OK".

Thanks Josh Keegan for requesting feature request!

The log rotate settings in system configuration has been supported

In the previous versions, log rotation options were supported as --log-rotate-age or --log-rotate-size via command line options.

On Windows, as Fluentd is launched as a windows service, it is required to configure again via --reg-winsvc-fluentdopt or edit fluentdopt registry key for log rotate customization.

These approach are not convenient for Windows users, so it may be better to support more comprehensive solution - customize via configuration file like this:

<system>
  <log>
    rotate_age 5
    rotate_size 1048576
  </log>
</system>

Now you can customize via rotate_age and rotate_size in log section.

fluent-cat: the issue resending secondary file in specific format has been fixed

The way to resend secondary is explained as How to resend secondary file, but it does not work if the record contains Fluent::EventTime. It raises Input must be a map (got Array) error.

In this release, it supports such a secondary file too.

Note that fluent-cat ignore timestamp which is stored in secondary records. (It is intended behavior for keeping consistency)

Miscellaneous bug fixes

  • Fixed to disable trace_instruction when RubyVM::InstructionSequence is available. It improves compatibility with truffleruby some extent. #3376
  • in_tail: Safely skip files which are used by another process on Windows. It improves exception handling about ERROR_SHARING_VIOLATION on Windows. #3378

Community topic

About three months has passed since community forum was moved to Discuss. Fluentd category is used actively nowadays! Note that fluentd forum on google groups is already deprecated.

Enjoy logging!

Read More

Fluentd v1.12.4 has been released

Hi users!

We have released v1.12.4. This release is a maintenance release of v1.12 series. ChangeLog is here.

in_tail: Fix a bug that refresh_watcher fails to handle file rotations

In the previous Fluentd v1.11.x implementation uses path based tailing keys. Since v1.12, follow_inodes feature is introduced, but as a side effects unexpectedly, refresh_watcher tend to fail to handle file rotations correctly.

This version contains a fix for the issue only. #3393

Enjoy logging!

Read More

FluentCon 2021 Recap & Updates

We’ve just had a series of amazing sessions from leaders and friends in the Fluentd & Fluent Bit community from Chronosphere, Calyptia, Nieman and Marcus, Cisco, Microsoft, Amazon, SAP, Zendesk and many community members & attendees from all over the world. There were variety of use cases, production challenges and solutions, performance optimization with great benchmark results at scale. Hope you enjoyed the conference and here are some notes for your review.

1st FluentCon co-located with KubeCon

1st FluentCon, co-located with KubeCon Europe, had a variety of topics covered starting from Fluentd & Fluent Bit updates including native metrics support and followed by great talks.

Observability for operation at scale

We really felt the vendor neutral observability is getting more momentum and exciting activities in "Tracing", "Metrics" and "Logs" are coming together to bring the operation at scale to the next level.

Fluentd and Fluent Bit Updates

  1. Fluentd turns 10 in June! Happy 10th birthday!
  2. Fluent Bit is being deployed over 2 million times a day.
  3. Conrainer deployments are 10x higher than package deployments.
  4. New release of Fluent Bit 1.7 and Fluentd 1.12
    • Fluent Bit multi-workers, new SSL library, and improved I/O disk performance (Available now!)

More Transparency, Accessibility and Neutrality for Observability

  1. New ways to engage the community with https://discuss.fluentd.org

  2. Building for more Vendor Nertality with additional plugins.

    • New native support for Metric handling (Available now!)
    • Native integration with Prometheus (Available now!)
    • Stream Processor support to Extract Metrics from Log data (Coming soon)
    • Support for Open Telemetry Protocol (OLTP) (Coming later this year.)

Simpler Plugin Development

  1. New WebAssembly integration with Fluent Bit (Coming later this year.)
  2. Write plugins the language you want. Keep the blazing fast performance.

Thanks to the foundation and conference speakers and sponsors

We would like to thank Cloud Native Computing Foundation and conference oganizer team for all the community effort making people meet and discuss in this challenging moment. We also thank all the amazing speakers and sponsors to support the 1st FluentCon. Here is the list of speakers and sponsors.

  1. Opening Remarks and Keynote

    • Martin Mao, CEO/Co-founder at Chronosphere
    • Anurag Gupta, Product Management at Calyptia
  2. Fluent Bit - Swiss Army Tool of Observability Data Ingestion

  3. Exporting your trace telemetry with OpenTelemetry and Fluent Bit

    • Aditya Jagdishkumar Prajapati
  4. Logging Operator the Cloud Native Fluent Ecosystem

  5. Lightning Talk: Scaling the Fluent Bit Kubernetes Filter in very large clusters

    • Drew Zhang & Nithish Kumar Murcherla, Amazon
  6. Lightning Talk: Fluentd as syslog drain for CloudFoundry

    • Karsten Schnitter, SAP SE
  7. Lightning Talk: Fluentd at Scale; Keys to successful Logging Syndication

    • Fred Moyer, SRE Observability Economist, Zendesk
  8. Lightning Talk: Parsing CRI JSON logs with Fluent Bit

  9. Customized Cloud Native Logging: Fluent Bit in Open Service Mesh

    • Sanya Kochhar, Software Engineer, Azure at Microsoft
  10. Scaling Log Collection with Fluent Bit

    • Wesley Pettit & Ugur Kira, Amazon
  11. Noise Cancelling Headphones for Fluent Bit - Powered by Lua

    • Matt Lehman at GreyNoise Intelligence Inc.
  12. Keynote: DevSecOps in the US Air Force and Closing Remarks

    • Nicolas Chaillan, Air Force Chief Software Office, US Air Force
    • Anurag Gupta & Eduardo Silva, Calyptia

Platinum Sponsor : Fluentd Subscription Network by ITOCHU Techno-Solution America, Inc.

Read More

Fluentd v1.12.3 has been released

Hi users!

We have released v1.12.3. ChangeLog is here.

TLSServer: Enable TCP keep-alive support for TLS connections

TLS implementation was updated to allow the use of send_keepalive_packet. This feature can be used to detect dropped TLS connections, and to prevent them from consuming too much system resource.

<source>
  @type forward
  send_keepalive_packet true
  <transport tls>
    cert_path /path/to/fluentd.crt
    ...
  </transport>
</source>

TLSServer: Show peer information on connection errors

Starting from this version, Fluentd will includes the client information when it fails to establish a secure connection, to ease the diagnosis of TLS-related problems (such as invalid client certs).

2021-04-23 17:35:03 +0900 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL host="127.0.0.1" port=34634 error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca

in_tail: Fix IO error handling on Windows system

Previously, in_tail had a bug that reports a bogus IO error code on the Windows platform (e.g. reports ERROR_SHARING_VIOLATION on missing files).

Starting from this version, in_tail will retrieve Windows error codes correctly, and show much more descriptive messages for them.

Fluent::Win32Error: code: 32, The process cannot access the file because it is being used by another process. - C:\file.txt",

in_tail: Handle short-lived log files correctly

Fluentd v1.12.2 had a race-condition bug that throws an uncatched exception on short-lived files #3327. This version contains a fix for the issue.

Miscellaneous bug fixes

  • Fix parser_type symbol conflicts in parser_csv and parser_syslog. #3302
  • Fix position file corruption issue on concurrent gracefulReloads #3335
  • Fix incorrect warnings about ${chunk_id} with out_s3 #3339

Enjoy logging!

Read More

Fluentd v1.12.2 has been released

Hi users!

We have released v1.12.2. ChangeLog is here.

out_copy: Add a new attribute ignore_if_prev_successes

You can start using ignore_if_prev_successes to define fallback outputs. Fluentd will make use of these destinations if (and only if) the preceding outputs failed.

<store ignore_error>
  @type test
  name c0
</store>
<store ignore_if_prev_success ignore_error>
  @type test
  name c1
</store>
<store ignore_if_prev_success>
  @type test
  name c2
</store>

time-format: Add a new option time_format_fallbacks

Fluentd is now able to handle a heterogeneous time field. For example, if your timestamp field is mostly unixtime, but sometimes formatted in iso8601, you can specify as follows:

time_type mixed
time_format_fallbacks unixtime, %iso8601

formatter_ltsv: Safe delimiter character handling

To prevent LTSV data from being corrupted, we've started to substitute a delimiter character in records with a safe replacement. You can fine-tune the behaviour using replacement option as follows:

<format>
  @type ltsv
  replacement " "
</format>

Several in_tail stability fixes

We’ve fixed several in_tail stability issues:

  • in_tail may not send rotated logs when mv is used to rotate #3292
  • in_tail crashes worker upon ENOENT #3274

out_forward: Fix duplication logs at shutdown

Fluentd now waits ack responses in the shutdown phase correctly #3137.

Project Maintenance

  • Our community forum has been moved to Discuss. Google group is deprecated.
  • We've migrated CI from Travis CI and AppVeyor to GitHub Actions #3281 #3290
  • We greet new maintainers from ClearCode Inc. #3282

Enjoy logging!

Read More

About Fluentd

Fluentd is an open source data collector to simplify log management.

Learn

Want to learn the basics of Fluentd? Check out these pages.

Ask the Community

Couldn't find enough information? Let's ask the community!

Ask the Experts

You need commercial-grade support from Fluentd committers and experts?

Follow Us!