Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes

TL;DR

This paper investigates the effectiveness of reinforcement learning methods in stochastic domains for language models, revealing that Group Relative Policy Optimization (GRPO) causes overconfidence, unlike other methods like PPO and RLOO.

Contribution

The study identifies that standard normalization in GRPO leads to overconfidence in stochastic outcomes and offers a theoretical explanation for this phenomenon.

Findings

GRPO induces overconfidence in stochastic outcomes

Removing normalization in GRPO improves calibration

PPO and RLOO produce well-calibrated models in stochastic domains

Abstract

Reinforcement learning (RL) has proven remarkably effective at improving the accuracy of language models in verifiable and deterministic domains like mathematics. Here, we examine if current RL methods are also effective at optimizing language models in verifiable domains with stochastic outcomes, like scientific experiments. Through applications to synthetic data and real-world biological experiments, we demonstrate that Group Relative Policy Optimization (GRPO) induces overconfident probability predictions for binary stochastic outcomes, while Proximal Policy Optimization (PPO) and REINFORCE Leave-One-Out (RLOO) yield well-calibrated models. We show that removing group standard normalization in GRPO fixes its miscalibration and provide a theoretical explanation for why normalization causes overconfidence. Our results provide new evidence against the use of standard normalization in GRPO and help pave the way for applications of RL for reasoning language models beyond deterministic domains.