FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization

TL;DR

FIPO introduces a reinforcement learning algorithm that enhances large language models' reasoning by using future-KL divergence for dense advantage calculation, leading to longer reasoning chains and improved accuracy.

Contribution

It proposes a novel dense advantage formulation using future-KL divergence, improving reasoning length and accuracy in large language models.

Findings

Extended chain-of-thought length from 4,000 to over 10,000 tokens.

Increased AIME 2024 Pass@1 accuracy from 50.0% to 58.0%.

Outperformed baseline models in reasoning tasks.

Abstract

We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0\%). This outperforms both DeepSeek-R1-Zero-Math-32B (around 47.0%) and o1-mini (approximately 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORM-based algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.