VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning

TL;DR

VinciCoder is a unified multimodal code generation model that uses a two-stage training process, including supervised fine-tuning and visual reinforcement learning, to improve performance across diverse benchmarks.

Contribution

The paper introduces VinciCoder, a novel model that unifies multimodal code generation with a coarse-to-fine visual reinforcement learning strategy, enhancing generalization and visual fidelity.

Findings

Achieves state-of-the-art results on multiple benchmarks.

The coarse-to-fine ViRL strategy significantly improves visual fidelity.

Large-scale dataset of 1.6M image-code pairs supports training.

Abstract

Multimodal code generation has garnered significant interest within the research community. Despite the notable success of recent vision-language models (VLMs) on specialized tasks like chart-to-code generation, their reliance on single-task training regimens fosters a narrow paradigm that hinders the development of generalized \textbf{VI}sio\textbf{N} \textbf{C}ode \textbf{I}ntelligence. In this work, we introduce \textbf{VinciCoder}, a unified multimodal code generation model that addresses this limitation via a two-stage training framework. We begin by constructing a large-scale Supervised Finetuning (SFT) corpus comprising 1.6M image-code pairs for tasks involving direct code generation and visual-based code refinement. Subsequently, we introduce a Visual Reinforcement Learning (ViRL) strategy, which employs a coarse-to-fine reward mechanism to improve visual fidelity by calculating visual similarity across local and global image patches. Extensive experiments on diverse multimodal code generation benchmarks demonstrate that VinciCoder achieves state-of-the-art performance, surpassing recent open-source models. The ablation study further validates the effectiveness of our proposed coarse-to-fine ViRL strategy. The data, code and model is available at https://github.com/DocTron-hub/VinciCoder.