Data Lake Ingestion Management

TL;DR

This paper presents a comprehensive approach to managing data ingestion in data lakes, including a metadata model, algorithms for ingestion, and a user-friendly metadata management system.

Contribution

It introduces a novel metadata model and algorithms specifically designed for effective data ingestion and management in data lakes.

Findings

Enhanced data findability and accessibility in data lakes.

Effective metadata management improves data reuse.

Algorithms ensure reliable data storage and metadata instantiation.

Abstract

Data Lake (DL) is a Big Data analysis solution which ingests raw data in their native format and allows users to process these data upon usage. Data ingestion is not a simple copy and paste of data, it is a complicated and important phase to ensure that ingested data are findable, accessible, interoperable and reusable at all times. Our solution is threefold. Firstly, we propose a metadata model that includes information about external data sources, data ingestion processes, ingested data, dataset veracity and dataset security. Secondly, we present the algorithms that ensure the ingestion phase (data storage and metadata instanciation). Thirdly, we introduce a developed metadata management system whereby users can easily consult different elements stored in DL.