KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes

TL;DR

Kramabench is a comprehensive benchmark designed to evaluate AI systems' ability to automate complex data-to-insight pipelines over data lakes, revealing current limitations in end-to-end pipeline generation.

Contribution

This paper introduces KramaBench, a new benchmark with curated challenges and an evaluation framework for assessing AI systems' capabilities in data pipeline automation.

Findings

Current AI systems achieve only 55% end-to-end accuracy.

Even with perfect data retrieval, accuracy peaks at 62%.

Leading LLMs identify 42% of data tasks but fully implement only 20%.

Abstract

Discovering insights from a real-world data lake potentially containing unclean, semi-structured, and unstructured data requires a variety of data processing tasks, ranging from extraction and cleaning to integration, analysis, and modeling. This process often also demands domain knowledge and project-specific insight. While AI models have shown remarkable results in reasoning and code generation, their abilities to design and execute complex pipelines that solve these data-lake-to-insight challenges remain unclear. We introduce KramaBench which consists of 104 manually curated and solved challenges spanning 1700 files, 24 data sources, and 6 domains. KramaBench focuses on testing the end-to-end capabilities of AI systems to solve challenges which require automated orchestration of different data tasks. KramaBench also features a comprehensive evaluation framework assessing the pipeline design and individual data task implementation abilities of AI systems. We evaluate 8 LLMs using our single-agent reference framework DS-Guru, alongside both open- and closed-source single- and multi-agent systems, and find that while current agentic systems may handle isolated data-science tasks and generate plausible draft pipelines, they struggle with producing working end-to-end pipelines. On KramaBench, the best system reaches only 55% end-to-end accuracy in the full data-lake setting. Even with perfect retrieval, the accuracy tops out at 62%. Leading LLMs can identify up to 42% of important data tasks but can only fully implement 20% of individual data tasks. Our code, reference framework, and data are available at https://github.com/mitdbg/KramaBench.