Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback

TL;DR

This paper introduces RRVF, a novel framework enabling multimodal models to learn complex visual reasoning solely from raw images by leveraging reinforcement learning and the verification of rendered outputs, reducing dependence on image-text supervision.

Contribution

The paper presents RRVF, a new reinforcement learning-based framework that allows visual reasoning models to learn from images alone, bypassing the need for curated image-text datasets.

Findings

Outperforms existing open-source MLLMs on image-to-code tasks

Demonstrates superior generalization across domains

Outperforms the more advanced MLLM used during training

Abstract

Multimodal Large Language Models (MLLMs) exhibit impressive performance across various visual tasks. Subsequent investigations into enhancing their visual reasoning abilities have significantly expanded their performance envelope. However, a critical bottleneck in the advancement of MLLMs toward deep visual reasoning is their heavy reliance on curated image-text supervision. To solve this problem, we introduce a novel framework, ``Reasoning-Rendering-Visual-Feedback'' (RRVF), that enables MLLMs to learn complex visual reasoning from only raw images. This framework builds on the ``Asymmetry of Verification'' principle, i.e., verifying the rendered output against the source image is substantially easier than performing deep visual reasoning to generate a faithful, structured representation such as code. We demonstrate that this relative ease provides an ideal reward signal for optimization via Reinforcement Learning (RL), thereby reducing reliance on image-text supervision. RRVF implements a closed-loop iterative process encompassing reasoning, rendering, and visual feedback components, enabling the model to perform complex reasoning, including self-correction through multi-turn interactions. This process is optimized end-to-end using the GRPO algorithm. Extensive evaluations are conducted on image-to-code generation across two diverse domains: data charts and web interfaces. The RRVF-trained model not only outperforms existing similarly sized open-source MLLMs and supervised fine-tuning baselines but also exhibits superior generalization. Notably, the model outperforms the more advanced MLLM used to generate visual feedback during training. Code is available at https://github.com/L-O-I/RRVF.