What is a Data Lake - The basics

In the next 10 years, the global generation of data will grow from 16 zettabytes, to 160 zettabytes, says an estimate by IDC. In addition to this, the forecast by Deloitte claims that unstructured data is set to grow at twice that rate, with the average financial institution accumulating 9 times more unstructured data than structured data by 2020. And it stands to reason that data generation by enterprises in every industry will increase in a similar fashion.

All this data is crucial for businesses - for understanding trends, formulating strategies, understanding customer behaviour and preferences, catering to those requirements and building new products and services. But actually gathering, storing and working with data is never an easy task. Yes, the sheer volume of data seems intimidating, but that’s the least of our problems.

The fact that data is stored fragmented, in silos across the organization, or that a lot of enterprise data is never used because it’s not in the right format are currently some of the biggest challenges for enterprise working with big data.

Solution? Data lake.

What is a Data Lake?

A data lake is a part of the data management system of an enterprise, designed to serve as a centralized repository for any data, of any size, in its raw and native format. The most important element to note here is that a data lake architecture can store unstructured and unorganized data in its natural form for later use. This data is tagged with multiple relevant markers so it’s easy to search with any related query.

Data lakes operate on the ELT strategy:

• Extract data from various sources like websites, mobile apps, social media etc
• Load data in the data lake, in its native format
• Transform it later to derive meaningful insights as and when there is a specific     business requirement.
  
  Since it is raw, the data can be transformed in the format of choice and convenience. When a business question arises, the data lake can be searched for relevant data sets which can be analyzed to help answer those questions. This is possible because the schema of the stored data are not defined in the repository, unless it is required by a business process.

This possibility of exploration and free association of unstructured data often leads to the discovery of more interesting insights than predicted.

How is Data Lake Different from a Data Warehouse 

A data Lake is often mistaken for a different version of a data warehouse. Though the basic function is the same – data storage, they both differ in the way information is stored in them.

Storing information in data warehouses requires properly defining the data, converting it into acceptable formats and defining its use case beforehand. In the process of data storage in a warehouse, the ‘transformation’ step of the ELT strategy comes before the ‘Loading’ phase. With a data warehouse:

• Data is always structured and organized before being stored

• Sources of data collection are limited

• Data usage may be limited to a few pre-defined operational purposes and it may not be possible to exploit it to its highest potential

What are the Advantages of a Data Lake Architecture

Given the fact that enterprises collect huge volumes of data is different systems across the organization, a data lake can go a long way in helping leverage it all. Some of the key reasons to build a data lake are:

• Diverse sources: Generally, data repositories can accept data from limited sources, after it has been cleaned and transformed. Unlike those, data lakes store data from a large range of data sources like social media, IoT devices, mobile apps etc. This is irrespective of the structure and format of the data, and ensures that data from any business system is available for usage, whenever required.
• Ease of access to data: Not only does a data lake store information coming from various sources; it also makes it available for anyone in need of required data. Any business system can query the data lake for the right data, and define how it is processed and transformed to derive specific insights.
• Security: Although anyone can freely access any data in the lake, access to the information about the source of that data can be restricted. This makes any data exploitation, beyond requirement, very difficult.
• Ease of usage of data: The unprocessed data stored directly from the source allows greater freedom of usage to the information seeker. Data scientists and business systems working with the data do not need to adhere to a specific format while working with the data.
• Cost effective: Data lakes are a single platform, cost effective solution for storing large data coming from various sources within and outside the organization. Because a data lake is capable of storing all kinds of data, and easily scalable to accommodate growing volumes, it is a one-time investment for enterprises to get it in place. Integrating a data lake with your cloud is another option which allows you to control your cost as you only pay for the space you actually use.
• Analytics: Data lake architecture, when integrated with enterprise search and analytics techniques, can help firms derive insights from the vast structured and unstructured data stored. A data lake is capable of utilizing large quantities of coherent data along with deep learning algorithms to identify information that powers real-time advanced analytics. Processing raw data is very useful for machine learning, predictive analysis and data profiling.

Data Lake Use Cases

With the sheer variety and volume of data being stored, data lakes can be leveraged for a variety of use cases. A few of the most impactful ones would be:

Marketing Data Lake

The increasing focus on customer experience and personalization in marketing has data at the heart of it. Customer information, whether anonymized or personal, forms the base for understanding and personalizing for the user. Coupled with data on customer activity on the website, social media, transactions etc, it allows enterprise marketing teams to know and predict what their customers need.

With a marketing data lake, enterprises can gather data from external and internal systems and drop it all in one place. The possibilities with this data can be at several levels:

• Basic analytics can help get a comprehensive look into persona profiles and campaign performance
• Unstructured data coming from disparate sources can be queried and leverage to form basic and advanced personalization and recommendation engines for users
• Moving further, a 360 degree view of individual customers can be formed with a data lake, pulling together information on customer journey, preferences, social media activity, sentiment analysis and more. Because of the sheer diversity of data, it is possible to drill down into any aspect of the customer lifecycle
• Beyond this, enterprises can have data scientists perform exploratory analysis, look at the wide spectrum of data available, build some statistical models and check if any new patterns and insights emerge.

Cyber Security

Securing business information and assets is a crucial requirement for enterprises. This means cyber security data collection and analysis has to be proactive and always on. All such data can be constantly collected in data lakes, given its ability to store undefined data. It can also be constantly or periodically analyzed in order to identify any anomalies and their causes, to spot and nullify cyber threats in time.

Log Analytics

A lot of enterprises today rely on IoT data streaming in from various devices. A data lake can be the perfect storage solution to house this continuously expanding data stream. Teams also run quick cleaning processes on it and make it available for analysis across different business functions.

So that was a quick look at what is a data lake and why enterprises should consider building one. Moving forward, we’ll dive into how exactly to set up a data lake and the different levels of maturity for enterprise data lakes.

Interested in exploring how a data lake fits into your enterprise infrastructure? Talk to our expert team, and let’s find out how Srijan can help.