LamPO: A Lambda Style Policy Optimization for Reasoning Language Models

TL;DR

LamPO introduces a pairwise advantage approach for reinforcement learning in reasoning language models, improving credit assignment and training stability over existing methods.

Contribution

It proposes LamPO, a lambda-style policy optimization method that enhances reward comparison granularity and incorporates auxiliary rewards for better reasoning model training.

Findings

LamPO outperforms GRPO and recent RLVR methods on multiple reasoning benchmarks.

It achieves more stable training dynamics and improved sample efficiency.

Experimental results demonstrate consistent performance gains across various models and datasets.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving reasoning language models on tasks such as mathematics, coding, and scientific question answering. However, widely used group-relative objectives, such as GRPO, summarize each sampled group with scalar statistics and therefore discard fine-grained relational information among candidate responses. This weakens credit assignment under sparse outcome rewards, especially when multiple generated solutions differ only subtly in reasoning quality. We propose \textbf{LamPO}, a \textbf{Lambda-Style Policy Optimization} method that replaces scalar group advantages with a \emph{Pairwise Decomposed Advantage}. LamPO aggregates pairwise reward gaps within each response group and modulates each comparison by a confidence-aware weight computed from sequence log-probability differences, while retaining the critic-free and clipped-update structure of PPO-style optimization. When reference solutions are available, we further add a lightweight ROUGE-L-based dense auxiliary reward to reduce reward sparsity. Experiments on AIME24, AIME25, MATH-500, and GPQA-Diamond with Qwen3-1.7B, Qwen3-4B, and Phi-4-mini show that LamPO consistently improves over GRPO and recent RLVR variants, with more stable training dynamics and better sample efficiency.