Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation

TL;DR

This paper introduces Multimodal Structured Reinforcement Learning (MSRL), a novel approach that combines textual and visual feedback to significantly improve chart-to-code generation, surpassing the limitations of supervised fine-tuning.

Contribution

It proposes MSRL with a multi-granularity reward system and a two-stage curriculum training strategy, achieving state-of-the-art results on large-scale real-world datasets.

Findings

MSRL improves performance by over 6% on ChartMimic.

MSRL outperforms existing methods in chart-to-code tasks.

Large-scale dataset of 3 million chart-code pairs enhances training.

Abstract

While reinforcement learning (RL) has proven highly effective for general reasoning in vision-language models, its application to tasks requiring deep understanding of information-rich images and structured output generation remains underexplored. Chart-to-code generation exemplifies this challenge, demanding complex reasoning over visual charts to produce structured code. Supervised fine-tuning (SFT) alone is often insufficient, highlighting the need for effective RL strategies tailored to structured outputs. In this paper, we systematically investigate the performance plateau of SFT through large-scale experiments and propose Multimodal Structured Reinforcement Learning (MSRL) for chart-to-code generation. We construct the largest training corpus to date, with 3 million chart-code pairs curated from real-world tables in arXiv papers, addressing the limitations of previous synthetic datasets. Despite achieving state-of-the-art performance, our experiments show that simply increasing SFT data eventually leads to diminishing improvements. To break this plateau, MSRL employs a multi-granularity reward system that integrates both textual and visual feedback. At the textual level, rule-based rewards validate fine-grained code details, while at the visual level, a model-based reward assesses the structural similarity between rendered code and ground-truth charts. We implement a two-stage curriculum training strategy, first optimizing the model with textual rewards and then incorporating visual signals for further enhancement. Experimental results demonstrate that MSRL substantially breaks the SFT plateau, improving high-level metrics by 6.2% and 9.9% on ChartMimic and ReachQA benchmarks, respectively. Notably, our method outperforms all existing approaches in the chart domain and achieves competitive results with advanced closed-source models.