In todays world where we are bound to use the resource at the optimum level by giving high performance. New logging architecture for EFK suffest that we should use the combination of both fluentBit and FluentD.
Let me expain you in details what I means to say you. In general pratice we want our microservice to run in Kubernetes enviroment and this Microservice will produce logs. As log beat is for logstash we have fluentbit for fluentD. AS you know Fluend is good agreegator with lots of plugin to be presice more than 3000 plugin input, output, filter etc. How ever it is resource consuming entity. Where as fluentbit is light weight and can be installed as an agent or deamonset or sidecar proxy with out Microservice. FluentBit is useful as a log collector as it did not consume much resource from our eco system.
In belwo example i had given you a simple approach
Spring boot application –> Send log to FluentBit –> furhter this fluentbit send log to FluentD –> finally we see the log in console of both Fluentbit and fluentD
In real scenario you need to send log from microservice to fluent-bit and then to fluentD and finally need to inject into Elastic search.
Make sure you have both fluentBit and FluentD installed in your machine. I am using windows so installed fluentbit and fluentd and put them into your path.
I had created mine own Springboot microservice which will create log inside mine below folder
logging.file.name=C:/springboot-log/spring-boot-eflk.log
Create two conf file
1- fluentbit-to-fluentd.conf :- This file will be used by our fluentbit while starting
[SERVICE]
# Flush
# =====
# set an interval of seconds before to flush records to a destination
flush 5
# Daemon
# ======
# instruct Fluent Bit to run in foreground or background mode.
daemon Off
# Log_Level
# =========
# Set the verbosity level of the service, values can be:
#
# - error
# - warning
# - info
# - debug
# - trace
#
# by default 'info' is set, that means it includes 'error' and 'warning'.
log_level info
# Parsers File
# ============
# specify an optional 'Parsers' configuration file
parsers_file parsers.conf
# Plugins File
# ============
# specify an optional 'Plugins' configuration file to load external plugins.
plugins_file plugins.conf
# HTTP Server
# ===========
# Enable/Disable the built-in HTTP Server for metrics
http_server Off
http_listen 0.0.0.0
http_port 2020
# Storage
# =======
# Fluent Bit can use memory and filesystem buffering based mechanisms
#
# - https://docs.fluentbit.io/manual/administration/buffering-and-storage
#
# storage metrics
# ---------------
# publish storage pipeline metrics in '/api/v1/storage'. The metrics are
# exported only if the 'http_server' option is enabled.
#
storage.metrics on
# storage.path
# ------------
# absolute file system path to store filesystem data buffers (chunks).
#
# storage.path /tmp/storage
# storage.sync
# ------------
# configure the synchronization mode used to store the data into the
# filesystem. It can take the values normal or full.
#
# storage.sync normal
# storage.checksum
# ----------------
# enable the data integrity check when writing and reading data from the
# filesystem. The storage layer uses the CRC32 algorithm.
#
# storage.checksum off
# storage.backlog.mem_limit
# -------------------------
# if storage.path is set, Fluent Bit will look for data chunks that were
# not delivered and are still in the storage layer, these are called
# backlog data. This option configure a hint of maximum value of memory
# to use when processing these records.
#
# storage.backlog.mem_limit 5M
[INPUT]
Name tail
Path C:/springboot-log/spring-boot-eflk.log
[OUTPUT]
Name forward
Match *
Host 192.168.0.14
Port 24224
[OUTPUT]
Name stdout
Match *
2- fluentd.conf :- This will be used by our FluentD while starting.
<source>
@type forward
bind 192.168.0.14
port 24224
tag *
</source>
<match *>
type stdout
</match>
Step 1:- start our spring boot applicaiton and check if we are able to create our log in side given folder.
Step 2:- Start our FluentD using belwo parameters
C:\fluend-conf-files>fluentd -c C:\fluend-conf-files\fluentd.conf
C:\fluend-conf-files>fluentd -c C:\fluend-conf-files\fluentd.conf
2021-06-07 17:29:23 +0530 [info]: parsing config file is succeeded path="C:\\fluend-conf-files\\fluentd.conf"
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-kafka' version '0.16.1'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-parser-winevt_xml' version '0.2.2'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-prometheus' version '1.8.5'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-td' version '1.1.0'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-06-07 17:29:23 +0530 [info]: gem 'fluent-plugin-windows-eventlog' version '0.8.0'
2021-06-07 17:29:23 +0530 [info]: gem 'fluentd' version '1.12.3'
2021-06-07 17:29:23 +0530 [warn]: 'type' is deprecated parameter name. use '@type' instead.
2021-06-07 17:29:23 +0530 [info]: using configuration file: <ROOT>
<source>
@type forward
bind "192.168.0.14"
port 24224
tag "*"
</source>
<match *>
type stdout
</match>
</ROOT>
2021-06-07 17:29:23 +0530 [info]: starting fluentd-1.12.3 pid=4684 ruby="2.7.3"
2021-06-07 17:29:23 +0530 [info]: spawn command to main: cmdline=["C:/opt/td-agent/bin/ruby.exe", "-Eascii-8bit:ascii-8bit", "C:/opt/td-agent/bin/fluentd", "-c", "C:\\fluend-conf-files\\fluentd.conf", "--under-supervisor"]
2021-06-07 17:29:26 +0530 [warn]: #0 'type' is deprecated parameter name. use '@type' instead.
2021-06-07 17:29:26 +0530 [info]: adding match pattern="*" type="stdout"
2021-06-07 17:29:26 +0530 [info]: adding source type="forward"
2021-06-07 17:29:27 +0530 [info]: #0 starting fluentd worker pid=12392 ppid=4684 worker=0
2021-06-07 17:29:27 +0530 [info]: #0 listening port port=24224 bind="192.168.0.14"
2021-06-07 17:29:27 +0530 [info]: #0 fluentd worker is now running worker=0Step 3:- Start our fluentBit using below command
C:\fluent-bit-conf-files>fluent-bit -c C:\fluent-bit-conf-files\fluentbit-to-fluentd.conf -R C:\fluent-bit-conf-files\parsers.conf
C:\fluent-bit-conf-files>fluent-bit -c C:\fluent-bit-conf-files\fluentbit-to-fluentd.conf -R C:\fluent-bit-conf-files\parsers.conf
Fluent Bit v1.7.4
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2021/06/07 17:15:06] [error] [parser] parser named 'apache' already exists, skip.
[2021/06/07 17:15:06] [ info] [engine] started (pid=1792)
[2021/06/07 17:15:06] [ info] [storage] version=1.1.1, initializing...
[2021/06/07 17:15:06] [ info] [storage] in-memory
[2021/06/07 17:15:06] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/06/07 17:15:06] [ info] [sp] stream processor started
Now lets hit the url
check our fluentBit log it will show our log in the console.
21/06/07 17:15:06] [ info] [sp] stream processor started
[0] tail.0: [1623066311.893832500, {"log"=>"2021-06-07 17:15:11.847 INFO 12456 --- [http-nio-9898-exec-8] ngbootDockerKubernetesExampleApplication : response found : Simple data message showing success call :- Mon Jun 07 17:15:11 IST 2021"}]
Now check the out put of fluentD console
2021-06-07 17:29:33.529573600 +0530 *: {"log":"2021-06-07 17:29:33.439 INFO 12456 --- [http-nio-9898-exec-7] ngbootDockerKubernetesExampleApplication : response found : Simple data message showing success call :- Mon Jun 07 17:29:33 IST 2021"}
You can see that we had forwarded our microservice log to fluenD using fluentbit.
you can download springboot example using below link
https://github.com/shdhumale/efk-springboot-docker-kubernetes-example.git
you can download conf file from below given location
shdhumale 
Gosh, man, thanks a lot!
I’ve been trying 6 hours to get my fluentbit => fluentd forwarding work and the problem was I was using “output tcp” in fluentbit and “input forwarding” in fluentd, and none of this crap showed any error.
LikeLike
YWC 🙂
LikeLike