ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering

TL;DR

ChartReasoner introduces a novel two-stage framework that converts charts into structured code for precise, interpretable reasoning, significantly improving visual reasoning in chart question answering tasks.

Contribution

The paper presents a new code-driven approach for visual reasoning over charts, including a high-fidelity chart-to-code conversion model and a scalable data synthesis pipeline for training.

Findings

Achieves performance comparable to state-of-the-art models with fewer parameters.

Effectively preserves chart details for accurate reasoning.

Demonstrates strong results on four public benchmarks.

Abstract

Recently, large language models have shown remarkable reasoning capabilities through long-chain reasoning before responding. However, how to extend this capability to visual reasoning tasks remains an open challenge. Existing multimodal reasoning approaches transfer such visual reasoning task into textual reasoning task via several image-to-text conversions, which often lose critical structural and semantic information embedded in visualizations, especially for tasks like chart question answering that require a large amount of visual details. To bridge this gap, we propose ChartReasoner, a code-driven novel two-stage framework designed to enable precise, interpretable reasoning over charts. We first train a high-fidelity model to convert diverse chart images into structured ECharts codes, preserving both layout and data semantics as lossless as possible. Then, we design a general chart reasoning data synthesis pipeline, which leverages this pretrained transport model to automatically and scalably generate chart reasoning trajectories and utilizes a code validator to filter out low-quality samples. Finally, we train the final multimodal model using a combination of supervised fine-tuning and reinforcement learning on our synthesized chart reasoning dataset and experimental results on four public benchmarks clearly demonstrate the effectiveness of our proposed ChartReasoner. It can preserve the original details of the charts as much as possible and perform comparably with state-of-the-art open-source models while using fewer parameters, approaching the performance of proprietary systems like GPT-4o in out-of-domain settings.