On data lake architectures and metadata management

TL;DR

This paper reviews data lake architectures and metadata management, clarifying their roles and challenges in handling large, diverse data sources for effective big data management.

Contribution

It provides a comprehensive overview of data lake design approaches, focusing on architectures and metadata management, and clarifies misconceptions about data lakes.

Findings

Analyzes various data lake architectures.

Highlights metadata management challenges.

Discusses pros and cons of data lakes.

Abstract

Over the past two decades, we have witnessed an exponential increase of data production in the world. So-called big data generally come from transactional systems, and even more so from the Internet of Things and social media. They are mainly characterized by volume, velocity, variety and veracity issues. Big data-related issues strongly challenge traditional data management and analysis systems. The concept of data lake was introduced to address them. A data lake is a large, raw data repository that stores and manages all company data bearing any format. However, the data lake concept remains ambiguous or fuzzy for many researchers and practitioners, who often confuse it with the Hadoop technology. Thus, we provide in this paper a comprehensive state of the art of the different approaches to data lake design. We particularly focus on data lake architectures and metadata management, which are key issues in successful data lakes. We also discuss the pros and cons of data lakes and their design alternatives.