Cloud Native Computing Foundation Internship at Fluent Bit: Weekly Blogs

Introduction

Hey everyone ! I am Atibhi Agrawal, a pre-final year student of Integrated Masters (Bachelors+Masters) in Electronics and Communication engineering course at International Institute of Information Technology, Bangalore. I will be contributing to fluent-d as a part of the CommunityBridge and Cloud Native Computing Foundation’s internship program. I will also be gettings 4 credits for the same as I have taken it as a project elective under Professor B.Thangaraju who takes the Operating Systems, Software Engineering courses etc. in our college.

How I heard of fluent-d

I was an intern at Hackerrank last summer where one of the tasks involved sending logs to stackdriver using fluent-d. That was the first time I heard of fluent-d. Fluentd is an open source data collector for unified logging layer. In November of 2020 I got the opportunity to go to KubeCon and Cloud Native Con North America as a diversity scholar. At the conference, I met Eduardo at the maintainers meetup. He told me about the internship and I applied for it. The application process consisted of tasks as well as a video interview with Eduardo and Masoud. I was so happy and grateful when I heard back from them that I was selected.

Week 1

January 6 to January 12 was the first week of the internship. I had a video call meeting with Eduardo and Masoud at the beginning of the week where we talked about the work that was expected of me. They also explained the general guidelines and code of conduct to be followed for the internship.

Currently, Fluent-bit exposes an API which is used by the plugins to read configuration values. This works as expected but problems arise when the plugin receives an unknown configuration key. There is also a need to reduce risk in case of bad configuration keys or depreciated property names. My task is to add config maps to plugin so that in the future dynamic reload support can be implemented. Dynamic reload is important so that the core is aware of the expected configuration properties of each plugin and it can perform validations before taking any reload action. You can refer to this issue for more details.

During the past week, I worked on adding config maps to nats and out_tcp plugin. My Pull Requests are under review. I also revised up on C basics and went through the codebase to get familiar with the code.

Plans for next week

I plan to add config maps for the remaining plugins in the next week.

I am really excited for my internship and look forward to contributing, learning and growing ! :smile:

Week 2

I registered config maps for 3 output plugins and made changes to the 2 PRs opened last week(the Pull Requests are under review.)

The plugins are :

1) out_azure

2) out_datadog

3) td

4) out_tcp

5) nats

I devised a rough workflow for migrating the plugins :

Test Plugin→ Add config map →Remove/Simplify code in Config File of plugin →Test Plugin→ Send PR → Make changes to PR

I also got my hands dirty with Microsoft Azure and Datadog to test if the plugins are working properly. This was particularly exciting for me because I am getting to learn about DevOps along with Software Engineering.

Moreover, I learnt a lot from my mentors Eduardo and Masoud. They explained in detail what the function of a config_map is and why offsetof is used in config maps. Writing down the explanation below for future reference.

Config Map

The config map has two main jobs:

1) Specify “which” parameters are allowed in any plugin.

2) Optional support to write the recieved parameter to the plugin context

3) When a config map is specified, if a parameter was not registered it will fail and exit.

offsetof()

The macro offsetof() returns the offset of the field member from the start of the structure type. offsetof() returns the offset of the given member within the given type, in units of bytes.

Plan for the next week

Week 3

This week I worked on the forward output plugin. It is the protocol used by fluentd to route messages between peers. It also provides interoperability between Fluentd and Fluentbit. I faced some issues while setting up Fluentd because the port that was used by Fluentd was also being used by ruby. After, I got the setup, I registered a config map for the forward plugin. However,forward output supports upstream servers and hence I had to read about Upstream Servers to test the plugin. While reading about upstream server I also read about TLS which is a widely adopted security protocol designed to facilitate privacy and data security for communications over the Internet. In fact, HTTPS is an implementation of TLS encryption on top of the HTTP protocol.

I also worked on the gelf and retry output plugins and registered config maps for them.

I also started working on splunk plugin but faced issues while setting up splunk in the cloud and sending logs to it. I shall complete this in the next few days after taking help from my mentors and also work on other output plugins.

Plan for the next week

Week 4

Work done -

Things learnt :) -

Plans for next week

Week 5

Work done -

This week I worked on the output plugins retry, forward and tcp. I made changes in the PR for these plugins as suggested by Eduardo. I learnt a lot about config_maps in general this week. I am going to summarize whatever I learnt :

Plans for next week

Week 6

Work done -

Plans for next week

Week 7

Work done

Plans for next week

Week 8

Work done

Pull Requests Waiting for Review

Plans for next week

Week 9

Work done

Plans for next week

Week 10

Work done

Plans for next half week

Week 11

Work done

Week 12

Work done

rss facebook twitter github youtube mail spotify lastfm instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora quora