Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoning

TL;DR

EAPO introduces a reward co-evolution approach to improve evidence retrieval and reasoning in long-context RL applications, significantly outperforming existing methods across multiple benchmarks.

Contribution

The paper proposes a novel Evidence-Augmented Policy Optimization framework with reward co-evolution, addressing reward sparsity and evidence extraction challenges in long-context reasoning.

Findings

EAPO outperforms SOTA baselines on eight benchmarks.

Reward model refinement improves evidence quality during training.

Adaptive co-evolution enhances long-context reasoning accuracy.

Abstract

While Reinforcement Learning (RL) has advanced LLM reasoning, applying it to long-context scenarios is hindered by sparsity of outcome rewards. This limitation fails to penalize ungrounded "lucky guesses," leaving the critical process of needle-in-a-haystack evidence retrieval largely unsupervised. To address this, we propose EAPO (Evidence-Augmented Policy Optimization). We first establish the Evidence-Augmented Reasoning paradigm, validating via Tree-Structured Evidence Sampling that precise evidence extraction is the decisive bottleneck for long-context reasoning. Guided by this insight, EAPO introduces a specialized RL algorithm where a reward model computes a Group-Relative Evidence Reward, providing dense process supervision to explicitly improve evidence quality. To sustain accurate supervision throughout training, we further incorporate an Adaptive Reward-Policy Co-Evolution mechanism. This mechanism iteratively refines the reward model using outcome-consistent rollouts, sharpening its discriminative capability to ensure precise process guidance. Comprehensive evaluations across eight benchmarks demonstrate that EAPO significantly enhances long-context reasoning performance compared to SOTA baselines.