Create Benchmarks for Data Lakes

TL;DR

This paper introduces a comprehensive benchmarking framework for data lakes, addressing the lack of standardized evaluation tools by covering diverse data types and workloads, and enabling fair comparison of different systems.

Contribution

The paper presents a new extensible and reproducible benchmark specifically designed for data lakes, including multiple data types and workload models not covered by existing benchmarks.

Findings

Benchmark effectively compares commercial and open-source data lakes

Performance metrics include query time and metadata handling

Demonstrates applicability on CloudLab platform

Abstract

Data lakes have emerged as a flexible and scalable solution for storing and analyzing large volumes of heterogeneous data, including structured, semi-structured, and unstructured formats. Despite their growing adoption in both industry and academia, there is a lack of standardized and comprehensive benchmarks for evaluating the performance of data lake systems. Existing benchmarks primarily target traditional data warehouses and focus on structured SQL workloads, making them insufficient for capturing the diverse workloads and access patterns typical of data lakes. In this work, we propose a new benchmarking framework for data lakes that aims to provide an objective and comparative evaluation of different data lake implementations. Our benchmark covers multiple data types and workload models, including data retrieval, aggregation, querying, and similarity search, which is a common yet underexplored operation in existing benchmarks. We measure key performance metrics such as query execution time, metadata generation time, and metadata size across different scale factors. The benchmark is designed to be extensible and reproducible, enabling users to generate datasets and evaluate data lake systems under realistic and diverse scenarios. We conduct our experiments on CloudLab and demonstrate how the proposed benchmark can be used to compare both commercial and open-source data lake platforms.