Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning

TL;DR

This paper introduces CalibRL, a hybrid-policy RLVR framework that enables controllable exploration in multi-modal reasoning models, improving training stability and performance by balancing exploration and exploitation through expert-guided mechanisms.

Contribution

CalibRL is a novel hybrid-policy RLVR method that uses distribution-aware advantage weighting and asymmetric activation to control exploration, addressing challenges of entropy collapse and policy degradation.

Findings

CalibRL achieves consistent improvements across eight benchmarks.

The framework effectively balances exploration and exploitation.

It stabilizes training by mitigating distributional mismatch.

Abstract

Reinforcement Learning with verifiable rewards (RLVR) has emerged as a primary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the enormous state space of MLLM and sparse rewards often leads to entropy collapse, policy degradation, or over-exploitation of suboptimal behaviors. This necessitates an exploration strategy that maintains productive stochasticity while avoiding the drawbacks of uncontrolled random sampling, yielding inefficient exploration. In this paper, we propose CalibRL, a hybrid-policy RLVR framework that supports controllable exploration with expert guidance, enabled by two key mechanisms. First, a distribution-aware advantage weighting scales updates by group rareness to calibrate the distribution, therefore preserving exploration. Meanwhile, the asymmetric activation function (LeakyReLU) leverages the expert knowledge as a calibration baseline to moderate overconfident updates while preserving their corrective direction. CalibRL increases policy entropy in a guided manner and clarifies the target distribution by estimating the on-policy distribution through online sampling. Updates are driven by these informative behaviors, avoiding convergence to erroneous patterns. Importantly, these designs help alleviate the distributional mismatch between the model's policy and expert trajectories, thereby achieving a more stable balance between exploration and exploitation. Extensive experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements, validating the effectiveness of our controllable hybrid-policy RLVR training. Code is available at https://github.com/zhh6425/CalibRL.