Skip-Connected Policy Optimization for Implicit Advantage

TL;DR

This paper introduces Skip-Connected Policy Optimization (SKPO), a novel RL method that decomposes reasoning into phases with skip connections, improving performance on mathematical and reasoning benchmarks.

Contribution

SKPO is a new approach that combines upstream dense reward reasoning with downstream group-relative optimization via skip connections, outperforming existing methods.

Findings

SKPO achieves 3.91% and 6.17% relative gains on key benchmarks.

SKPO produces higher intermediate-step quality trajectories.

SKPO outperforms strong baselines on out-of-domain reasoning tasks.

Abstract

Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness.