On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization

TL;DR

This paper investigates the negative gradient effects in group relative deep reinforcement learning, identifies the cause of Lazy Likelihood Displacement, and proposes NTHR to improve model performance on reasoning tasks.

Contribution

It provides a theoretical analysis of GRPO's learning dynamics, introduces NTHR to mitigate LLD, and demonstrates improved reasoning performance across multiple models.

Findings

NTHR effectively reduces Lazy Likelihood Displacement.

Models with NTHR show consistent performance gains on math reasoning benchmarks.

Theoretical analysis links LLD to naive token penalization in GRPO.

Abstract

Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs), with Group Relative Policy Optimization (GRPO) emerging as a widely used algorithm in recent systems. Despite GRPO's widespread adoption, we identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training. This behavior mirrors a recently discovered misalignment issue in Direct Preference Optimization (DPO), attributed to the influence of negative gradients. We provide a theoretical analysis of GRPO's learning dynamic, identifying the source of LLD as the naive penalization of all tokens in incorrect responses with the same strength. To address this, we develop a method called NTHR, which downweights penalties on tokens contributing to the LLD. Unlike prior DPO-based approaches, NTHR takes advantage of GRPO's group-based structure, using correct responses as anchors to identify influential tokens. Experiments on math reasoning benchmarks demonstrate that NTHR effectively mitigates LLD, yielding consistent performance gains across models ranging from 0.5B to 3B parameters.