Are complicated loss functions necessary for teaching LLMs to reason?

TL;DR

This paper investigates the necessity of complex loss components in training LLMs for reasoning, finding that simpler methods like REINFORCE with group advantage can outperform more complicated approaches like GRPO.

Contribution

The paper introduces RGRA, a simplified reinforcement learning approach that removes unnecessary components from GRPO, improving reasoning performance in LLMs.

Findings

Negative feedback is crucial for effective training.

PPO style constraints are not necessary for reasoning improvement.

RGRA outperforms GRPO on mathematical benchmarks.

Abstract

Recent advances in large language models (LLMs) highlight the importance of post training techniques for improving reasoning and mathematical ability. Group Relative Policy Optimization (GRPO) has shown promise in this domain by combining group relative advantage estimation, PPO style clipping, and KL regularization. However, its complexity raises the question of whether all components are necessary for fostering reasoning behaviors. We conduct a systematic analysis of GRPO and identify two key findings: (1) incorporating negative feedback is essential training solely on actions above a baseline limits learning; and (2) PPO style constraints, such as policy ratio clipping, are not required to improve mathematical reasoning or performance. Building on these insights, we propose REINFORCE with Group Relative Advantage (RGRA), a simplified variant that retains group relative advantage estimation but removes PPO style clipping and policy ratio terms. Experiments across standard mathematical benchmarks indicate that RGRA has the potential to achieve stronger performance than GRPO. Our results suggest that simpler REINFORCE based approaches can effectively enhance reasoning in LLMs, offering a more transparent and efficient alternative to GRPO.