GRPO and Reflection Reward for Mathematical Reasoning in Large Language Models

TL;DR

This paper introduces a four-stage framework combining Group Relative Policy Optimization (GRPO) with reflection rewards to enhance mathematical reasoning in large language models, achieving state-of-the-art results.

Contribution

It proposes a novel training framework that proactively encourages reflection in LLMs, improving their reasoning capabilities beyond existing methods.

Findings

GRPO achieves state-of-the-art performance in mathematical reasoning tasks.

Reflection rewards significantly improve reasoning accuracy.

Full-parameter supervised fine-tuning outperforms low-rank adaptation.

Abstract

The enhancement of reasoning capabilities in large language models (LLMs) has garnered significant attention, with supervised fine-tuning (SFT) and reinforcement learning emerging as dominant paradigms. While recent studies recognize the importance of reflection in reasoning processes, existing methodologies seldom address proactive reflection encouragement during training. This study focuses on mathematical reasoning by proposing a four-stage framework integrating Group Relative Policy Optimization (GRPO) with reflection reward mechanisms to strengthen LLMs' self-reflective capabilities. Besides, this approach incorporates established accuracy and format reward. Experimental results demonstrate GRPO's state-of-the-art performance through reflection-encouraged training, with ablation studies confirming the reflection reward's pivotal role. Comparative evaluations demonstrate full-parameter SFT's superiority over low-rank adaptation (LoRA) despite heightened computational demands. Building on these cumulative findings, this research substantiates GRPO's methodological significance in post-training optimization and envisions its potential to serve as a pivotal enabler for future LLM-based intelligent agents through the synergistic integration of cognitive rewards with dynamic environmental interactions.