IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis

TL;DR

IDABench introduces a multi-round benchmark for evaluating LLMs in interactive data analysis, revealing current models' limitations in iterative reasoning and decision-making compared to human experts.

Contribution

This work presents IDA-Bench, a new benchmark for assessing LLMs in multi-round data analysis tasks, emphasizing the importance of iterative reasoning capabilities.

Findings

State-of-the-art agents succeed on less than 50% of tasks

Current models struggle with multi-round iterative analysis

Highlights need for improved multi-turn reasoning in LLMs

Abstract

Large Language Models (LLMs) show promise as data analysis agents, but existing benchmarks overlook the iterative nature of the field, where experts' decisions evolve with deeper insights of the dataset. To address this, we introduce IDA-Bench, a novel benchmark evaluating LLM agents in multi-round interactive scenarios. Derived from complex Kaggle notebooks, tasks are presented as sequential natural language instructions by an LLM-simulated user. Agent performance is judged by comparing its final numerical output to the human-derived baseline. Initial results show that even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on < 50% of the tasks, highlighting limitations not evident in single-turn tests. This work underscores the need to improve LLMs' multi-round capabilities for building more reliable data analysis agents, highlighting the necessity of achieving a balance between instruction following and reasoning.