# Data Lake Organization

## TL;DR

This paper introduces a probabilistic model and an algorithm for organizing data lakes to enhance user navigation and data discovery, outperforming existing methods and supporting both navigation and keyword search.

## Contribution

It presents a novel probabilistic model and an approximate algorithm for optimizing data lake organization to improve user navigation and discovery.

## Key findings

- The proposed algorithm outperforms hand-curated taxonomies.
- Navigation helps users find tables not reachable by keyword search.
- Both navigation and keyword search are valued by users, serving as complementary tools.

## Abstract

We consider the problem of creating a navigation structure that allows a user to most effectively navigate a data lake. We define an organization as a graph that contains nodes representing sets of attributes within a data lake and edges indicating subset relationships among nodes. We present a new probabilistic model of how users interact with an organization and define the likelihood of a user finding a table using the organization. We propose the data lake organization problem as the problem of finding an organization that maximizes the expected probability of discovering tables by navigating an organization. We propose an approximate algorithm for the data lake organization problem. We show the effectiveness of the algorithm on both real data lakes containing data from open data portals and on benchmarks that emulate the observed characteristics of real data lakes. Through a formal user study, we show that navigation can help users discover relevant tables that cannot be found by keyword search. In addition, in our study, 42% of users preferred the use of navigation and 58% preferred keyword search, suggesting these are complementary and both useful modalities for data discovery in data lakes. Our experiments show that data lake organizations take into account the data lake distribution and outperform an existing hand-curated taxonomy and a common baseline organization.