ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding

TL;DR

ChartSketcher introduces a multimodal, feedback-driven reasoning approach for chart understanding, enabling iterative visual annotation and refinement to improve comprehension and accuracy in complex data visualizations.

Contribution

It proposes ChartSketcher, a novel model that uses visual annotations and reinforcement learning to enhance chart understanding beyond existing text-based reasoning methods.

Findings

Achieves promising results on chart understanding benchmarks.

Enables iterative visual reasoning with annotations.

Improves interpretability and accuracy in complex visual tasks.

Abstract

Charts are high-density visualization carriers for complex data, serving as a crucial medium for information extraction and analysis. Automated chart understanding poses significant challenges to existing multimodal large language models (MLLMs) due to the need for precise and complex visual reasoning. Current step-by-step reasoning models primarily focus on text-based logical reasoning for chart understanding. However, they struggle to refine or correct their reasoning when errors stem from flawed visual understanding, as they lack the ability to leverage multimodal interaction for deeper comprehension. Inspired by human cognitive behavior, we propose ChartSketcher, a multimodal feedback-driven step-by-step reasoning method designed to address these limitations. ChartSketcher is a chart understanding model that employs Sketch-CoT, enabling MLLMs to annotate intermediate reasoning steps directly onto charts using a programmatic sketching library, iteratively feeding these visual annotations back into the reasoning process. This mechanism enables the model to visually ground its reasoning and refine its understanding over multiple steps. We employ a two-stage training strategy: a cold start phase to learn sketch-based reasoning patterns, followed by off-policy reinforcement learning to enhance reflection and generalization. Experiments demonstrate that ChartSketcher achieves promising performance on chart understanding benchmarks and general vision tasks, providing an interactive and interpretable approach to chart comprehension.