Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning

TL;DR

This paper introduces RGR-GRPO, a rubric-driven reinforcement learning framework that enhances multi-domain reasoning in large language models by providing dense rewards and offline guidance, leading to significant performance improvements.

Contribution

The paper presents RGR-GRPO, a novel RL framework leveraging rubrics for multi-domain reasoning, enabling better exploration and outperforming existing methods.

Findings

RGR-GRPO outperforms baseline RL methods across 14 benchmarks.

Achieves +7.0% to +8.4% improvements in key reasoning tasks.

Maintains stable entropy and enhances pass@k performance.

Abstract

Recent advances in reinforcement learning (RL) have significantly improved the complex reasoning capabilities of large language models (LLMs). Despite these successes, existing methods mainly focus on single-domain RL (e.g., mathematics) with verifiable rewards (RLVR), and their reliance on purely online RL frameworks restricts the exploration space, thereby limiting reasoning performance. In this paper, we address these limitations by leveraging rubrics to provide both fine-grained reward signals and offline guidance. We propose $\textbf{RGR-GRPO}$ (Reward and Guidance through Rubrics), a rubric-driven RL framework for multi-domain reasoning. RGR-GRPO enables LLMs to receive dense and informative rewards while exploring a larger solution space during GRPO training. Extensive experiments across 14 benchmarks spanning multiple domains demonstrate that RGR-GRPO consistently outperforms RL methods that rely solely on alternative reward schemes or offline guidance. Compared with verifiable online RL baseline, RGR-GRPO achieves average improvements of +7.0%, +5.4%, +8.4%, and +6.6% on mathematics, physics, chemistry, and general reasoning tasks, respectively. Notably, RGR-GRPO maintains stable entropy fluctuations during off-policy training and achieves superior pass@k performance, reflecting sustained exploration and effective breakthrough beyond existing performance bottlenecks.