We’ve just had a series of amazing sessions from leaders and friends in the Fluentd & Fluent Bit community from Chronosphere, Calyptia, Nieman and Marcus, Cisco, Microsoft, Amazon, SAP, Zendesk and many community members & attendees from all over the world. There were variety of use cases, production challenges and solutions, performance optimization with great benchmark results at scale. Hope you enjoyed the conference and here are some notes for your review.
1st FluentCon, co-located with KubeCon Europe, had a variety of topics covered starting from Fluentd & Fluent Bit updates including native metrics support and followed by great talks.
We really felt the vendor neutral observability is getting more momentum and exciting activities in "Tracing", "Metrics" and "Logs" are coming together to bring the operation at scale to the next level.
New ways to engage the community with https://discuss.fluentd.org
Building for more Vendor Nertality with additional plugins.
We would like to thank Cloud Native Computing Foundation and conference oganizer team for all the community effort making people meet and discuss in this challenging moment. We also thank all the amazing speakers and sponsors to support the 1st FluentCon. Here is the list of speakers and sponsors.
Fluent Bit - Swiss Army Tool of Observability Data Ingestion
Exporting your trace telemetry with OpenTelemetry and Fluent Bit
Logging Operator the Cloud Native Fluent Ecosystem
Lightning Talk: Scaling the Fluent Bit Kubernetes Filter in very large clusters
Lightning Talk: Fluentd as syslog drain for CloudFoundry
Lightning Talk: Fluentd at Scale; Keys to successful Logging Syndication
Lightning Talk: Parsing CRI JSON logs with Fluent Bit
Customized Cloud Native Logging: Fluent Bit in Open Service Mesh
Scaling Log Collection with Fluent Bit
Noise Cancelling Headphones for Fluent Bit - Powered by Lua
Keynote: DevSecOps in the US Air Force and Closing Remarks
Platinum Sponsor : Fluentd Subscription Network by ITOCHU Techno-Solution America, Inc.
Hi users!
We have released v1.12.3. ChangeLog is here.
TLSServer: Enable TCP keep-alive support for TLS connectionsTLS implementation was updated to allow the use of send_keepalive_packet. This feature can be used to detect dropped TLS connections, and to prevent them from consuming too much system resource.
<source>
@type forward
send_keepalive_packet true
<transport tls>
cert_path /path/to/fluentd.crt
...
</transport>
</source>
TLSServer: Show peer information on connection errorsStarting from this version, Fluentd will includes the client information when it fails to establish a secure connection, to ease the diagnosis of TLS-related problems (such as invalid client certs).
2021-04-23 17:35:03 +0900 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL host="127.0.0.1" port=34634 error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 state=error: tlsv1 alert unknown ca
in_tail: Fix IO error handling on Windows systemPreviously, in_tail had a bug that reports a bogus IO error code on the Windows platform (e.g. reports ERROR_SHARING_VIOLATION on missing files).
Starting from this version, in_tail will retrieve Windows error codes correctly, and show much more descriptive messages for them.
Fluent::Win32Error: code: 32, The process cannot access the file because it is being used by another process. - C:\file.txt",
in_tail: Handle short-lived log files correctlyFluentd v1.12.2 had a race-condition bug that throws an uncatched exception on short-lived files #3327. This version contains a fix for the issue.
parser_type symbol conflicts in parser_csv and parser_syslog. #3302${chunk_id} with out_s3 #3339Enjoy logging!
Hi users!
We have released v1.12.2. ChangeLog is here.
out_copy: Add a new attribute ignore_if_prev_successesYou can start using ignore_if_prev_successes to define fallback outputs. Fluentd will make use of these destinations if (and only if) the preceding outputs failed.
<store ignore_error>
@type test
name c0
</store>
<store ignore_if_prev_success ignore_error>
@type test
name c1
</store>
<store ignore_if_prev_success>
@type test
name c2
</store>
time_format_fallbacksFluentd is now able to handle a heterogeneous time field. For example, if your timestamp field is mostly unixtime, but sometimes formatted in iso8601, you can specify as follows:
time_type mixed
time_format_fallbacks unixtime, %iso8601
formatter_ltsv: Safe delimiter character handlingTo prevent LTSV data from being corrupted, we've started to substitute a delimiter character in records with a safe replacement. You can fine-tune the behaviour using replacement option as follows:
<format>
@type ltsv
replacement " "
</format>
in_tail stability fixesWe’ve fixed several in_tail stability issues:
out_forward: Fix duplication logs at shutdownFluentd now waits ack responses in the shutdown phase correctly #3137.
Enjoy logging!
Hi users!
We have released v1.12.1. ChangeLog is here.
out_http: A new option headers_from_placeholders addedYou can start using headers_from_placeholders to embed tags and record fields into HTTP headers.
headers_from_placeholders {"x-foo-bar":"${$.foo.bar}","x-tag":"app-${tag}"}
<buffer tag,$.foo.bar>
# buffer parameters...
</buffer>
fluent-plugin-config-format is now able to show the list of configuration options using a Markdown table.
$ fluent-plugin-config-format -t input dummy
...
### Configuration
|parameter|type|description|default|
|---|---|---|---|
|tag|string (required)|The value is the tag assigned to the generated events.||
|size|integer (optional)|The number of events in event stream of each emits.|`1`|
|rate|integer (optional)|It configures how many events to generate per second.|`1`|
|auto_increment_key|string (optional)|If specified, each generated event has an auto-incremented key field.||
|suspend|bool (optional)|The boolean to suspend-and-resume incremental value after restart<br>Deprecated: This parameters is ignored||
|sample| (optional)|The sample data to be generated. An array of JSON hashes or a single JSON hash.<br>Alias: dummy|`[{"message"=>"sample"}]`|
disable_shared_socket addedYou can now prevent Fluentd from creating a communication socket by setting disable_shared_socket option (or --disable-shared-socket command-line parameter).
This option is useful, in particular, on Windows when you do not want Fluentd from occupying an ephemeral TCP port. Read the documentation for details.
Enjoy logging!
td-agent “v4” is available since August 2020. You' might’ve been wondering what the upgrade process is. You are in a right place, In this post, we will share the steps we’ve tested and hopefully this will help your experience from v3 to v4.
In the td-agent v4, core components like ruby(2.4 -> 2.7) and jemalloc(4.5.0 -> 5.2.1) were updated as well as removing libraries for 3rd party gems like postgreSQL to improve the maintainability of packages.
During the upgrade process, plugins bundled in td-agent are automatically upgraded. With that being said, other plugins added on your own are not included. You should review if you need to upgrade plugins since some directory structures from v3 and v4 are changed. In this post, I will show steps with plugins added on my own, “fluent-plugin-mongo“ for instance. Here is sample configuration file I used through steps.
[root@fluent02 ~]# cat /etc/td-agent/td-agent.conf
<source>
@type syslog
port 5140
bind 0.0.0.0
tag system
</source>
<match system.**>
@type copy
<store>
@type stdout
</store>
<store>
@type mongo
database rsyslog
collection system
host 127.0.0.1
port 27017
</store>
</match>
[root@fluent02 ~]# td-agent-gem list | grep fluent-plugin*
fluent-plugin-elasticsearch (4.0.9)
fluent-plugin-kafka (0.13.0)
fluent-plugin-mongo (1.5.0)
fluent-plugin-prometheus (1.8.0)
fluent-plugin-prometheus_pushgateway (0.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.3.0)
fluent-plugin-s3 (1.3.2)
fluent-plugin-systemd (1.0.2)
fluent-plugin-td (1.1.0)
fluent-plugin-td-monitoring (0.2.4)
fluent-plugin-webhdfs (1.2.5)
You can also find installed plugins under /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/ directories.
[root@fluent02 ~]# ll /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/ | grep fluent-plugin*
drwxrwxr-x. 5 root root 4096 Dec 2 06:17 fluent-plugin-elasticsearch-4.0.9
drwxrwxr-x. 3 root root 169 Dec 2 06:17 fluent-plugin-kafka-0.13.0
drwxr-xr-x. 5 root root 209 Dec 2 06:54 fluent-plugin-mongo-1.5.0
drwxrwxr-x. 5 root root 195 Dec 2 06:17 fluent-plugin-prometheus-1.8.0
drwxrwxr-x. 6 root root 206 Dec 2 06:17 fluent-plugin-prometheus_pushgateway-0.0.2
drwxrwxr-x. 3 root root 161 Dec 2 06:17 fluent-plugin-record-modifier-2.1.0
drwxrwxr-x. 3 root root 210 Dec 2 06:17 fluent-plugin-rewrite-tag-filter-2.3.0
drwxrwxr-x. 3 root root 222 Dec 2 06:17 fluent-plugin-s3-1.3.2
drwxrwxr-x. 3 root root 49 Dec 2 06:17 fluent-plugin-systemd-1.0.2
drwxrwxr-x. 3 root root 236 Dec 2 06:17 fluent-plugin-td-1.1.0
drwxrwxr-x. 4 root root 152 Dec 2 06:17 fluent-plugin-td-monitoring-0.2.4
drwxrwxr-x. 3 root root 176 Dec 2 06:17 fluent-plugin-webhdfs-1.2.5
[root@fluent02 ~]# systemctl stop td-agent
When RedHat, you can run following script.
[root@fluent02 ~]# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent4.sh | sh
You can find more information about the installation script in Fluend Doc - Installation.
[root@fluent02 ~]# yum info td-agent
Installed Packages
Name : td-agent
Version : 4.0.1
Release : 1.el8
Architecture : x86_64
Size : 59 M
Source : td-agent-4.0.1-1.el8.src.rpm
Repository : @System
From repo : treasuredata
Summary : The stable distribution of Fluentd
URL : https://www.treasuredata.com/
License : ASL 2.0
Description : The stable distribution of Fluentd, called td-agent.
[root@fluent02 ~]# systemctl daemon-reload
[root@fluent02 ~]# td-agent-gem list | grep fluent-plugin*
fluent-plugin-elasticsearch (4.1.1)
fluent-plugin-kafka (0.14.1)
fluent-plugin-prometheus (1.8.2)
fluent-plugin-prometheus_pushgateway (0.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.3.0)
fluent-plugin-s3 (1.4.0)
fluent-plugin-systemd (1.0.2)
fluent-plugin-td (1.1.0)
fluent-plugin-webhdfs (1.2.5)
You can see bundled plugins are upgraded as well but can not find plugins added on my own. In this post, added plugin was “fluent-plugin-mongo“ and it is not shown in installed list.
[root@fluent02 ~]# td-agent-gem install fluent-plugin-mongo
[root@fluent02 ~]# td-agent-gem list | grep fluent-plugin*
fluent-plugin-elasticsearch (4.1.1)
fluent-plugin-kafka (0.14.1)
fluent-plugin-mongo (1.5.0)
fluent-plugin-prometheus (1.8.2)
fluent-plugin-prometheus_pushgateway (0.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.3.0)
fluent-plugin-s3 (1.4.0)
fluent-plugin-systemd (1.0.2)
fluent-plugin-td (1.1.0)
fluent-plugin-webhdfs (1.2.5)
As for td-agent v4, “fluent-plugin-mongo“ was installed under “/opt/td-agent/lib/ruby/gems/2.7.0/gems/” directories.
[root@fluent02 ~]# ll /opt/td-agent/lib/ruby/gems/2.7.0/gems/ | grep fluent-plugin*
drwxr-xr-x. 5 root root 4096 Dec 2 08:26 fluent-plugin-elasticsearch-4.1.1
drwxr-xr-x. 3 root root 169 Dec 2 08:26 fluent-plugin-kafka-0.14.1
drwxr-xr-x. 5 root root 209 Dec 2 08:36 fluent-plugin-mongo-1.5.0
drwxr-xr-x. 4 root root 200 Dec 2 08:26 fluent-plugin-prometheus-1.8.2
drwxr-xr-x. 6 root root 206 Dec 2 08:26 fluent-plugin-prometheus_pushgateway-0.0.2
drwxr-xr-x. 3 root root 161 Dec 2 08:26 fluent-plugin-record-modifier-2.1.0
drwxr-xr-x. 3 root root 210 Dec 2 08:26 fluent-plugin-rewrite-tag-filter-2.3.0
drwxr-xr-x. 3 root root 222 Dec 2 08:26 fluent-plugin-s3-1.4.0
drwxr-xr-x. 3 root root 49 Dec 2 08:26 fluent-plugin-systemd-1.0.2
drwxr-xr-x. 3 root root 236 Dec 2 08:26 fluent-plugin-td-1.1.0
drwxr-xr-x. 3 root root 176 Dec 2 08:26 fluent-plugin-webhdfs-1.2.5
[root@fluent02 ~]# systemctl start td-agent
[root@fluent02 ~]# tail -100f /var/log/td-agent/td-agent.log
i) Restart “sshd“ daemon.
[root@fluent02 mongo]# systemctl restart sshd
ii) Confirm messages in “stdout“ output.
[root@fluent02 ~]# tail -100f /var/log/td-agent/td-agent.log
---------- Sample message ----------
2020-12-02 08:50:29.000000000 +0000 system.authpriv.info: {"host":"fluent02","ident":"sshd","pid":"2960","message":"Server listening on :: port 22."}
iii) Create sample script to read data from MongoDB.
[root@fluent02 mongo]# vim mongo_test.py
from pymongo import MongoClient
from bson.json_util import loads, dumps
import json
client = MongoClient('localhost', 27017)
db = client.rsyslog
collection = db.system
f = collection.find()
json_str = dumps(f)
print(json_str)
iv) Run sample script and confirm if same messages with “stdout“ are available.
[root@fluent02 mongo]# python3 mongo_test.py | jq
---------- Sample message ----------
{
"_id": {
"$oid": "5fc7559b0bf9f80b84137e1f"
},
"host": "fluent02",
"ident": "sshd",
"pid": "2960",
"message": "Server listening on :: port 22.",
"time": {
"$date": 1606899029000
}
}
}
Now, upgrading steps are completed. Happy Logging!
This blog was originally written by kubotat in the Fluentd Subscription Network.
Fluentd is an open source data collector to simplify log management.
2021-05-04: FluentCon 2021 Recap & Updates
2021-04-23: Fluentd v1.12.3 has been released
2021-03-29: Fluentd v1.12.2 has been released
2021-02-18: Fluentd v1.12.1 has been released
2021-02-01: Upgrade td-agent from v3 to v4
Want to learn the basics of Fluentd? Check out these pages.
Couldn't find enough information? Let's ask the community!
You need commercial-grade support from Fluentd committers and experts?
©2010-2021 Fluentd Project. ALL Rights Reserved.
Fluentd is a hosted project under the Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.