Learning Self-Correction in Vision-Language Models via Rollout Augmentation

TL;DR

This paper introduces Octopus, a reinforcement learning framework that enhances self-correction in vision-language models by synthesizing dense training examples, leading to improved performance and efficiency across multiple benchmarks.

Contribution

The paper presents Octopus, a novel RL rollout augmentation method with response-masking, enabling effective self-correction learning in large vision-language models.

Findings

Achieves state-of-the-art results on 7 benchmarks.

Outperforms RLVR baseline by 1.0 score.

Requires only 0.72x training time per step.

Abstract

Self-correction is essential for solving complex reasoning problems in vision-language models (VLMs). However, existing reinforcement learning (RL) methods struggle to learn it, as effective self-correction behaviors emerge only rarely, making learning signals extremely sparse. To address this challenge, we propose correction-specific rollouts (Octopus), an RL rollout augmentation framework that synthesizes dense self-correction examples by recombining existing rollouts. This augmentation simultaneously improves sample efficiency due to rollout reuse and stabilizes RL optimization through balanced supervision. Furthermore, we introduce a response-masking strategy that decouples self-correction from direct reasoning, avoiding signal conflicts and enabling both behaviors to be learned effectively. Building on this, we introduce Octopus-8B, a reasoning VLM with controllable self-correction capability. Across 7 benchmarks, it achieves SoTA performance among open-source VLMs, outperforming the best RLVR baseline by 1.0 score while requiring only $0.72\times$ training time per step.