UniDataBench: Evaluating Data Analytics Agents Across Structured and Unstructured Data

TL;DR

UniDataBench is a comprehensive benchmark designed to evaluate data analytics agents across diverse real-world data sources, enabling assessment of their ability to extract insights from structured and unstructured data.

Contribution

The paper introduces UniDataBench, a new benchmark for evaluating data analytics agents on multiple data formats, and proposes ReActInsight, a novel LLM-based autonomous analysis agent.

Findings

ReActInsight effectively discovers cross-source linkages.

UniDataBench covers a wide range of real-world datasets.

The framework advances data analytics capabilities in diverse scenarios.

Abstract

In the real business world, data is stored in a variety of sources, including structured relational databases, unstructured databases (e.g., NoSQL databases), or even CSV/excel files. The ability to extract reasonable insights across these diverse source is vital for business success. Existing benchmarks, however, are limited in assessing agents' capabilities across these diverse data types. To address this gap, we introduce UniDataBench, a comprehensive benchmark designed to evaluate the performance of data analytics agents in handling diverse data sources. Specifically, UniDataBench is originating from real-life industry analysis report and we then propose a pipeline to remove the privacy and sensitive information. It encompasses a wide array of datasets, including relational databases, CSV files to NoSQL data, reflecting real-world business scenarios, and provides unified framework to assess how effectively agents can explore multiple data formats, extract valuable insights, and generate meaningful summaries and recommendations. Based on UniDataBench, we propose a novel LLM-based agent named ReActInsight, an autonomous agent that performs end-to-end analysis over diverse data sources by automatically discovering cross-source linkages, decomposing goals, and generating robust, self-correcting code to extract actionable insights. Our benchmark and agent together provide a powerful framework for advancing the capabilities of data analytics agents in real-world applications.