Soft Sequence Policy Optimization

TL;DR

This paper introduces Soft Sequence Policy Optimization (SSPO), an off-policy RL method for LLM alignment that uses soft gating over token ratios, improving training stability and performance in reasoning tasks.

Contribution

The paper proposes SSPO, a novel off-policy sequence policy optimization method with soft gating, addressing stability and performance issues in LLM alignment.

Findings

SSPO enhances training stability in mathematical reasoning tasks.

SSPO outperforms existing methods in sequence-level reward optimization.

Theoretical analysis supports the effectiveness of soft gating in SSPO.

Abstract

A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we show that SSPO improves training stability and performance in mathematical reasoning tasks.