Metadata Systems for Data Lakes: Models and Features

TL;DR

This paper introduces MEDAL, a graph-based metadata model for data lakes, along with evaluation criteria, demonstrating its comprehensiveness over existing systems.

Contribution

The paper presents MEDAL, a novel, comprehensive graph-based metadata model for data lakes, and proposes evaluation criteria to assess metadata system effectiveness.

Findings

MEDAL outperforms existing metadata systems in comprehensiveness.

Evaluation criteria help assess metadata system effectiveness.

MEDAL's graph-based approach enhances metadata management.

Abstract

Over the past decade, the data lake concept has emerged as an alternative to data warehouses for storing and analyzing big data. A data lake allows storing data without any predefined schema. Therefore, data querying and analysis depend on a metadata system that must be efficient and comprehensive. However, metadata management in data lakes remains a current issue and the criteria for evaluating its effectiveness are more or less nonexistent.In this paper, we introduce MEDAL, a generic, graph-based model for metadata management in data lakes. We also propose evaluation criteria for data lake metadata systems through a list of expected features. Eventually, we show that our approach is more comprehensive than existing metadata systems.