Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards

TL;DR

This paper introduces PSN-RLVR, a parameter-space noise method for reinforcement learning with verifiable rewards, enhancing exploration and reasoning capabilities by perturbing policy parameters and employing adaptive noise scheduling.

Contribution

It proposes a novel parameter-space noise approach with a real-time adaptive scheduler, improving exploration and reasoning in RLVR over existing methods.

Findings

PSN-GRPO outperforms prior exploration methods on reasoning benchmarks.

Enhanced pass-at-k performance under large sampling budgets.

Method is orthogonal and composable with other RLVR techniques.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning, yet growing evidence indicates an exploration ceiling: it often reweights existing solution traces rather than discovering new strategies, limiting gains under large sampling budgets (e.g., pass-at-256). We address this limitation with PSN-RLVR, which perturbs policy parameters before rollout generation to induce temporally consistent, trajectory-level exploration that better preserves long-horizon chain-of-thought coherence than action-space noise. To mitigate the resulting sampling-update mismatch, we incorporate truncated importance sampling (TIS). To avoid expensive KL-based adaptive noise control, we propose a computationally efficient real-time adaptive noise scheduler driven by a lightweight surrogate that combines semantic diversity with normalized self-certainty. Instantiated on GRPO, a widely used RLVR method, PSN-GRPO consistently expands the effective reasoning capability boundary across multiple mathematical reasoning benchmarks and model families, yielding higher pass-at-k under large sampling budgets and outperforming prior exploration-oriented RLVR methods (e.g., Pass-at-k-style training) while remaining orthogonal and thus composable for additional gains.