Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents

TL;DR

This paper introduces the Data Agent Benchmark (DAB), a comprehensive evaluation framework for enterprise data agents that assesses their ability to integrate, transform, and analyze data across multiple heterogeneous systems, revealing current limitations.

Contribution

The paper presents DAB, the first benchmark evaluating the full data agent pipeline across diverse industries and systems, along with analysis of model performance and failure modes.

Findings

Best model achieves only 38% accuracy on DAB

Benchmark reveals significant challenges in data integration and transformation

Provides insights for future development of reliable data agents

Abstract

Users across enterprises increasingly rely on AI agents to query their data through natural language. However, building reliable data agents remains difficult because real-world data is often fragmented across multiple heterogeneous database systems, with inconsistent references and information buried in unstructured text. Existing benchmarks only tackle individual pieces of this problem -- e.g., translating natural-language questions into SQL queries, answering questions over small tables provided in context -- but do not evaluate the full pipeline of integrating, transforming, and analyzing data across multiple database systems. To fill this gap, we present the Data Agent Benchmark (DAB), grounded in a formative study of enterprise data agent workloads across six industries. DAB comprises 54 queries across 12 datasets, 9 domains, and 4 database management systems. On DAB, the best frontier model (Gemini-3-Pro) achieves only 38% pass@1 accuracy. We benchmark five frontier LLMs, analyze their failure modes, and distill takeaways for future data agent development. Our benchmark and experiment code are published at github.com/ucbepic/DataAgentBench.