Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generation

TL;DR

This paper introduces Chart Specification, a structured intermediate representation and reinforcement learning approach that significantly improves the accuracy and fidelity of chart-to-code generation by vision-language models, especially with limited training data.

Contribution

It proposes a novel structured representation and a reward mechanism for better structural fidelity in chart-to-code tasks, surpassing prior methods with less data.

Findings

Outperforms previous approaches on three benchmarks.

Achieves up to 61.7% improvement with only 3K training samples.

Establishes new state-of-the-art results with 4K samples.

Abstract

Vision-Language Models (VLMs) have shown promise in generating plotting code from chart images, yet achieving structural fidelity remains challenging. Existing approaches largely rely on supervised fine-tuning, encouraging surface-level token imitation rather than faithful modeling of underlying chart structure, which often leads to hallucinated or semantically inconsistent outputs. We propose Chart Specification, a structured intermediate representation that shifts training from text imitation to semantically grounded supervision. Chart Specification filters syntactic noise to construct a structurally balanced training set and supports a Spec-Align Reward that provides fine-grained, verifiable feedback on structural correctness, enabling reinforcement learning to enforce consistent plotting logic. Experiments on three public benchmarks show that our method consistently outperforms prior approaches. With only 3K training samples, we achieve strong data efficiency, surpassing leading baselines by up to 61.7% on complex benchmarks, and scaling to 4K samples establishes new state-of-the-art results across all evaluated metrics. Overall, our results demonstrate that precise structural supervision offers an efficient pathway to high-fidelity chart-to-code generation. Code and dataset are available at: https://github.com/Mighten/chart-specification-paper