Learning Deliberately, Acting Intuitively: Unlocking Test-Time Reasoning in Multimodal LLMs

TL;DR

This paper introduces the D2I framework that enhances multimodal LLM reasoning by using format rewards during training, enabling better reasoning without extra annotations, and shifting to intuitive reasoning during evaluation.

Contribution

The D2I method improves multimodal LLM reasoning by decoupling training and testing reasoning styles using format rewards, without requiring additional data annotations.

Findings

D2I outperforms baselines on various benchmarks.

Format reward fosters transferable reasoning skills.

Decoupling training and test reasoning enhances flexibility.

Abstract

Reasoning is a key capability for large language models (LLMs), particularly when applied to complex tasks such as mathematical problem solving. However, multimodal reasoning research still requires further exploration of modality alignment and training costs. Many of these approaches rely on additional data annotation and relevant rule-based rewards to enhance the understanding and reasoning ability, which significantly increases training costs and limits scalability. To address these challenges, we propose the Deliberate-to-Intuitive reasoning framework (D2I) that improves the understanding and reasoning ability of multimodal LLMs (MLLMs) without extra annotations and complex rewards. Specifically, our method sets deliberate reasoning strategies to enhance modality alignment only through the rule-based format reward during training. While evaluating, the reasoning style shifts to intuitive, which removes deliberate reasoning strategies during training and implicitly reflects the model's acquired abilities in the response. D2I outperforms baselines across both in-domain and out-of-domain benchmarks. Our findings highlight the role of format reward in fostering transferable reasoning skills in MLLMs, and inspire directions for decoupling training-time reasoning depth from test-time response flexibility.