Towards Schema Inference for Data Lakes

TL;DR

This paper presents a method for inferring schemas in data lakes by clustering data sets and identifying entity types and relationships, facilitating easier data discovery and interpretation.

Contribution

It introduces a novel schema inference approach using approximate indexes and clustering to identify entity types and their relationships in data lakes.

Findings

Effective in real-world data repositories

Improves data discovery and understanding

Highlights areas for further research

Abstract

A data lake is a repository of data with potential for future analysis. However, both discovering what data is in a data lake and exploring related data sets can take significant effort, as a data lake can contain an intimidating amount of heterogeneous data. In this paper, we propose the use of schema inference to support the interpretation of the data in the data lake. If a data lake is to support a schema-on-read paradigm, understanding the existing schema of relevant portions of the data lake seems like a prerequisite. In this paper, we make use of approximate indexes that can be used for data discovery to inform the inference of a schema for a data lake, consisting of entity types and the relationships between them. The specific approach identifies candidate entity types by clustering similar data sets from the data lake, and then relationships between data sets in different clusters are used to inform the identification of relationships between the entity types. The approach is evaluated using real-world data repositories, to identify where the proposal is effective, and to inform the identification of areas for further work.