PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning

TL;DR

PeRL introduces a permutation-enhanced reinforcement learning framework that significantly improves multimodal reasoning across multiple images by enhancing exploration and positional understanding, achieving state-of-the-art results.

Contribution

The paper proposes a novel permutation-based reinforcement learning approach with a multi-stage strategy for interleaved vision-language reasoning, addressing complex multi-image scenarios.

Findings

PeRL outperforms existing baselines on multi-image benchmarks.

Achieves state-of-the-art performance on 5 multi-image tasks.

Maintains competitive results on single-image benchmarks.

Abstract

Inspired by the impressive reasoning capabilities demonstrated by reinforcement learning approaches like DeepSeek-R1, recent emerging research has begun exploring the use of reinforcement learning (RL) to enhance vision-language models (VLMs) for multimodal reasoning tasks. However, most existing multimodal reinforcement learning approaches remain limited to spatial reasoning within single-image contexts, yet still struggle to generalize to more complex and real-world scenarios involving multi-image positional reasoning, where understanding the relationships across images is crucial. To address this challenge, we propose a general reinforcement learning approach PeRL tailored for interleaved multimodal tasks, and a multi-stage strategy designed to enhance the exploration-exploitation trade-off, thereby improving learning efficiency and task performance. Specifically, we introduce permutation of image sequences to simulate varied positional relationships to explore more spatial and positional diversity. Furthermore, we design a rollout filtering mechanism for resampling to focus on trajectories that contribute most to learning optimal behaviors to exploit learned policies effectively. We evaluate our model on 5 widely-used multi-image benchmarks and 3 single-image benchmarks. Our experiments confirm that PeRL trained model consistently surpasses R1-related and interleaved VLM baselines by a large margin, achieving state-of-the-art performance on multi-image benchmarks, while preserving comparable performance on single-image tasks.