Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning

TL;DR

This paper introduces ELPO, a method that improves tool-integrated reasoning in LLMs by localizing irrecoverable errors for better credit assignment, leading to significant performance gains across multiple benchmarks.

Contribution

ELPO is a novel approach that localizes irrecoverable errors in long-horizon reasoning tasks and leverages hierarchical advantage attribution for improved policy optimization.

Findings

ELPO outperforms strong RL baselines on TIR benchmarks.

ELPO improves Pass@K and Major@K scaling metrics.

ELPO enhances rollout ranking quality and tool-call efficiency.

Abstract

Tool-integrated reasoning (TIR) enables LLM agents to solve tasks through planning, tool use, and iterative revision, but outcome-only reinforcement learning in this setting suffers from sparse, delayed rewards and weak step-level credit assignment. In long-horizon TIR trajectories, an early irrecoverable mistake can determine success or failure, making it crucial to localize the first irrecoverable step and leverage it for fine-grained credit assignment. We propose Error-Localized Policy Optimization (ELPO), which localizes the first irrecoverable step via binary-search rollout trees under a fixed rollout budget, converts the resulting tree into stable learning signals through hierarchical advantage attribution, and applies error-localized adaptive clipping to strengthen corrective updates on the critical step and its suffix. Across TIR benchmarks in math, science QA, and code execution, ELPO consistently outperforms strong Agentic RL baselines under comparable sampling budgets, with additional gains in Pass@K and Major@K scaling, rollout ranking quality, and tool-call efficiency. Our code will be publicly released soon.