ChartM$^3$: Benchmarking Chart Editing with Multimodal Instructions

TL;DR

This paper introduces ChartM$^3$, a benchmark and training dataset for multimodal chart editing that combines natural language and visual cues, revealing current model limitations and improving capabilities through multimodal training.

Contribution

The paper presents a new benchmark, ChartM$^3$, for multimodal chart editing, and a large-scale training dataset to enhance model performance in interpreting combined visual and textual instructions.

Findings

Current multimodal models struggle with visual indicators.

Fine-tuning on ChartM$^3$-Train improves editing accuracy.

Benchmark enables comprehensive evaluation of multimodal chart editing.

Abstract

Charts are a fundamental visualization format widely used in data analysis across research and industry. While enabling users to edit charts based on high-level intentions is of great practical value, existing methods primarily rely on natural language instructions, which are often too ambiguous to support fine-grained editing. In this work, we introduce a novel paradigm for multimodal chart editing, where user intent is expressed through a combination of natural language and visual indicators that explicitly highlight the elements to be modified. To support this paradigm, we present Chart$\text{M}^3$, a new benchmark for Multimodal chart editing with Multi-level complexity and Multi-perspective evaluation. Chart$\text{M}^3$ contains 1,000 samples spanning four levels of editing difficulty. Each sample includes triplets in the form of (chart, code, multimodal instructions). To comprehensively evaluate chart editing models, Chart$\text{M}^3$ provides metrics that assess both visual appearance and code correctness. Our benchmark reveals significant limitations in current multimodal large language models (MLLMs), including GPT-4o, particularly in their ability to interpret and act on visual indicators. To address this, we construct Chart$\text{M}^3$-Train, a large-scale training set with 24,000 multimodal chart editing samples. Fine-tuning MLLMs on this dataset leads to substantial improvements, demonstrating the importance of multimodal supervision in building practical chart editing systems. Our datasets, codes, and evaluation tools are available at https://github.com/MLrollIT/ChartM3. %https://github.com/MLrollIT/ChartM3Our datasets, codes, and evaluation tools are available at https://github.com/yaolinli/VCE.