Play to Generalize: Learning to Reason Through Game Play

TL;DR

This paper introduces Visual Game Learning (ViGaL), a novel RL-based post-training method where multimodal large language models improve reasoning skills by playing arcade-like games, leading to better performance on diverse reasoning benchmarks.

Contribution

The paper presents a new RL post-training approach using gameplay to enhance reasoning in multimodal models, outperforming specialized models without additional supervised data.

Findings

Training on simple arcade games improves multimodal reasoning performance.

The model outperforms specialist models on reasoning benchmarks.

Gameplay-based training preserves general visual capabilities.

Abstract

Developing reasoning capabilities in multimodal large language models (MLLMs) remains challenging. Motivated by literature suggesting that gameplay promotes transferable reasoning skills, we propose a novel post-training method, Visual Game Learning (ViGaL), where MLLMs develop generalizable reasoning skills through playing arcade-like games. Specifically, we show that training a 7B-parameter MLLM via reinforcement learning (RL) on simple games like Snake significantly enhances the downstream performance on multimodal math benchmarks like MathVista, on multi-discipline questions like MMMU and on 3D spatial reasoning benchmarks like VSI-Bench, without seeing any worked solutions, equations, or diagrams during RL. Remarkably, our model outperforms specialist models post-trained on benchmark-oriented multimodal reasoning data, while preserving the model's performance on general visual benchmarks, a challenge where specialist models often fall short. Our findings suggest that multimodal reasoning can emerge from gameplay, pointing to a promising strategy of designing surrogate tasks for RL post-training.