Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

TL;DR

This paper introduces Listwise Policy Optimization (LPO), a novel reinforcement learning method for large language models that explicitly targets response distributions, improving training stability and diversity.

Contribution

LPO provides a new framework for RLVR that explicitly conducts target-projection on the response simplex, enhancing performance and stability over existing methods.

Findings

LPO achieves consistent improvements across diverse reasoning tasks.

LPO maintains optimization stability and response diversity.

LPO outperforms typical policy gradient baselines.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for large language models (LLMs) post-training to incentivize reasoning capacity. Among existing recipes, group-based policy gradient is prevalent, which samples a group of responses per prompt and updates the policy via group-relative advantage signals. This work reveals that these optimization strategies share a common geometric structure: each implicitly defines a target distribution on the response simplex and projects toward it via first-order approximation. Building on this insight, we propose Listwise Policy Optimization (LPO) to explicitly conduct the target-projection, which demystifies the implicit target by restricting the proximal RL objective to the response simplex, and then projects the policy via exact divergence minimization. This framework provides (i) monotonic improvement on the listwise objective with bounded, zero-sum, and self-correcting projection gradients, and (ii) flexibility in divergence selection with distinct structural properties through the decoupled projection step. On diverse reasoning tasks and LLM backbones, LPO consistently improves training performance over typical policy gradient baselines under matched targets, while intrinsically preserving optimization stability and response diversity.