EFK Kubernetes Stack in Cloud-Native Way — Scope About Retail Industry

What is EFK Stack?

“EFK” is an acronym for three major open source projects: Elasticsearch, Fluentd and Kibana. Elasticsearch is a search and analytics engine. Fluentd to receive, clean and parse the log data. Kibana lets users visualize data with charts and graphs in Elasticsearch.

Why EFK stack?

When Data constantly flow into your systems, it can quickly grow to be fat and stale. As it grows larger, your analytics will slow up, resulting in sluggish insights, which is likely to be a serious business problem. So, the BIG question for your Big Data is how can you maintain valuable business insights?

The solution is EFK stack which makes it way easier and faster to search and analyze large data sets. A detailed description of this EFK is given below.

Elasticsearch — The Amazing Log Search Tool:

Elasticsearch is an open-source, extensively distributable, promptly adaptable, web search tool that is available through a broad and expounds API. Elasticsearch can control incredibly quick searches that help your information revelation applications.

Elasticsearch is a powerful solution for your data extraction problems. A single developer can use it to find the needless data in the heap so that it saves both time and manpower.

Few Elasticsearch features are listed below:

1. Elasticsearch simplifies data ingest, visualization, and reporting.
2. It helps in fast and Incisive search against large volumes of data.
3. Real-time data and real-time analytics.
4. Elasticsearch comes with a wide set of features like scalability, high-availability and multi-tenant.

Fluentd — Routing Your Log Data:

Fluentd to collect, transform, and ship log data to the Elasticsearch backend. Fluentd is a popular open-source data collector that we’ll set up on our Kubernetes nodes to tail container log files, filter and transform the log data, and deliver it to the Elasticsearch cluster, where it will be indexed and stored.

Kibana — Visualizing Your Log Data:

Kibana is a data visualization and management tool for Elasticsearch that provides real-time histograms, line graphs, pie charts, and maps. Kibana also includes advanced applications such as Canvas, which allows users to create custom dynamic info graphics which are suitable for their data, and Elastic Maps for visualizing geospatial data.

EFK Stack Architecture

The data from the different sources are stored in the pods of each cluster, which are collected by Fluentd. Later it transforms and ships to Elasticsearch backend. In Elasticsearch data are analyzed and later it moves to the Kibana and there the visualization of data takes place. The customers are working with the Kibana dashboard and the rest of the applications are supported at the backend. Kibana dashboard is easy and user-friendly one, here the user can get different graphs according to user needs.

Fig.1.:Architecture of EFK Stack

EFK in Production-Environment

For those companies who are receiving thousands of logs coming on every second and they need a centralized and scalable solution that would allow them to search across these logs quickly, then EFK stack is the best solution.

1: E-Commerce Industry:

To build the new e-commerce platform, EFK stack was used to support the volume of transactions operated by stores on the platform. Below I have described How EFK stack is used in the E-commerce industry

2: Supply chain service Industry:

For supply chain services providers it helps to take a few decisions based on logs in less time which impacts cost, revenue, service and so on.

To maintain a high level of operations, these industries need to know everything that’s going on all the time. Every minute they capture the lot more data from the field, including the size, location, and status of all its shipments. It is difficult for them to analyze the data stored in databases like MongoDB or any other if the data size is bulk, so EFK stack helps them to solve all major issues regarding analyzing the data.

3: Insurance or finance companies:

Any Insurance or finance companies are no longer needs to write scripts or spend days to investigate IT events. They can gather and visualize the data they need and respond to events immediately. It reduced the complexity and cost of analytics and also Using Kibana has empowered everyone to analyze data, removing the need for data engineers to do the job.

How can we use the EFK stack to manage log data

Let us consider one example, here we took data of a company named GrabnPay.

GrabnPay India is a budget online store launched in India as a division of GrabnPay group UK. Over the years of its trustworthy services, Gradient satisfaction, and also the quality of products delivered to the shoppers, Grabnpay has evolved itself as darling alternative to stay themselves travel with the newest trends in fashion clothes, accessories, gifts and an exclusive collection of home & Kitchen accessories. The versatility and also the vary of products out there continuously build the shop distinctive.

This company is highly optimizing their weekly to monthly operations by considering the analysis constantly and reducing the expenses gradually in product returns from customers, inventory management, regional delivery. Log on to their website to know more about them.

They are facing issues while analyzing their data such as:

• How many orders dispatched per day?
• Which state has more number of orders?
• What is the revenue on a daily basis?
• How many orders got returned every day/week? and so on

They got an optimized solution to all their problems with one powerful tool named EFK stack. This tool helps them to analyze the data easily and quickly.

They are trying to analyses the city which got more number of orders, so that they can concentrate more on shipping and placement of orders and also they can take special attention on those cities where the order traffic is less. But analyzing manually causes more time consuming so EFK helps to overcome from this issue.

Fig.2.: Pie chart shows the list of orders received from the cities

The above pie chart shows the cities which got the highest number of orders, where Kolkata ranks first and Secunderabad at last. This helps the user to analyze easily and graphical representation helps them to make decisions regarding the business fast. It depends on the user according to their convenience they can choose the chart type.

Fig.3.: Pie chart shows the orders received from the states

The above pie chart shows the states from where the orders are received.

They were facing another issue regarding the cancellation of the order. When they analyzed it with the help of EFK, they got to know that the cancellation of the order percentage is more, which was more due to the lack of communication between the customers. Later they started to interact with the customers, and find a solution to this problem. This reduces the issue and now the cancellation of order percentage is about 3.23%.

Fig.4.: Pie chart shows the order confirmations

Summary:

The EFK stack finds a major role in the log analytics sector. The retail industry, e-commerce, healthcare services and most of the companies where the data are bulk and analyzing them takes a lot of time and effort finds this as an amazing tool. Kibana helps them to visualize the data in a graphical representation which makes the user easily understand the logs and helps them in taking effective business ideas.

For more details about cloud-native Microservices & cloud-native stack transformation, please refer to Yobitel Communications.