ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models

TL;DR

ChartEditBench is a new benchmark designed to evaluate the multi-turn, grounded chart editing capabilities of multimodal language models, highlighting their strengths and limitations in iterative data visualization tasks.

Contribution

It introduces a comprehensive benchmark with a robust evaluation framework for multi-turn chart editing, addressing limitations of prior one-shot assessments.

Findings

State-of-the-art models struggle with multi-turn context maintenance.

Models perform well on stylistic edits but often fail on data transformations.

Error accumulation significantly impacts multi-turn editing performance.

Abstract

While Multimodal Large Language Models (MLLMs) perform strongly on single-turn chart generation, their ability to support real-world exploratory data analysis remains underexplored. In practice, users iteratively refine visualizations through multi-turn interactions that require maintaining common ground, tracking prior edits, and adapting to evolving preferences. We introduce ChartEditBench, a benchmark for incremental, visually grounded chart editing via code, comprising 5,000 difficulty-controlled modification chains and a rigorously human-verified subset. Unlike prior one-shot benchmarks, ChartEditBench evaluates sustained, context-aware editing. We further propose a robust evaluation framework that mitigates limitations of LLM-as-a-Judge metrics by integrating execution-based fidelity checks, pixel-level visual similarity, and logical code verification. Experiments with state-of-the-art MLLMs reveal substantial degradation in multi-turn settings due to error accumulation and breakdowns in shared context, with strong performance on stylistic edits but frequent execution failures on data-centric transformations. ChartEditBench, establishes a challenging testbed for grounded, intent-aware multimodal programming.