From Sight to Insight: Improving Visual Reasoning Capabilities of Multimodal Models via Reinforcement Learning

TL;DR

This paper enhances multimodal models' visual reasoning by using reinforcement learning with reward functions to promote structured reasoning and visual understanding, significantly improving performance on visual tasks.

Contribution

It introduces reward-driven reinforcement learning to improve visual reasoning in open-source multimodal models without costly supervision.

Findings

Converting images to text boosts reasoning performance by over 23%.

Reward functions improve structured reasoning and visual understanding.

Experiments show 5.56% performance gains on Qwen-2.5-VL-7B.

Abstract

Reinforcement learning (RL) has emerged as a promising approach for eliciting reasoning chains before generating final answers. However, multimodal large language models (MLLMs) generate reasoning that lacks integration of visual information. This limits their ability to solve problems that demand accurate visual perception, such as visual puzzles. We show that visual perception is the key bottleneck in such tasks: converting images into textual descriptions significantly improves performance, yielding gains of 26.7% for Claude 3.5 and 23.6% for Claude 3.7. To address this, we investigate reward-driven RL as a mechanism to unlock long visual reasoning in open-source MLLMs without requiring costly supervision. We design and evaluate six reward functions targeting different reasoning aspects, including image understanding, thinking steps, and answer accuracy. Using group relative policy optimization (GRPO), our approach explicitly incentivizes longer, structured reasoning and mitigates bypassing of visual information. Experiments on Qwen-2.5-VL-7B achieve 5.56% improvements over the base model, with consistent gains across both in-domain and out-of-domain settings.