OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning

TL;DR

OGER introduces a unified offline-guided exploration reward for reinforcement learning that enhances reasoning capabilities and out-of-domain generalization in large language models.

Contribution

It presents a novel framework combining offline guidance and online RL with entropy-aware rewards, improving reasoning and exploration.

Findings

OGER outperforms baselines on reasoning benchmarks.

Achieves significant gains in mathematical reasoning tasks.

Maintains robust generalization to out-of-domain tasks.

Abstract

Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have significantly improved Large Language Model (LLM) reasoning, yet models often struggle to explore novel trajectories beyond their initial latent space. While offline teacher guidance and entropy-driven strategies have been proposed to address this, they often lack deep integration or are constrained by the model's inherent capacity. In this paper, we propose OGER, a novel framework that unifies offline teacher guidance and online reinforcement learning through a specialized reward modeling lens. OGER employs multi-teacher collaborative training and constructs an auxiliary exploration reward that leverages both offline trajectories and the model's own entropy to incentivize autonomous exploration. Extensive experiments across mathematical and general reasoning benchmarks demonstrate that OGER significantly outperforms competitive baselines, achieving substantial gains in mathematical reasoning while maintaining robust generalization to out-of-domain tasks. We provide a comprehensive analysis of training dynamics and conduct detailed ablation studies to validate the effectiveness of our entropy-aware reward modulation. Our code is available at https://github.com/ecoli-hit/OGER.git.