PiCA: Pivot-Based Credit Assignment for Search Agentic Reinforcement Learning

TL;DR

PiCA introduces a pivot-based reward mechanism for search agents in reinforcement learning, effectively addressing long-horizon credit assignment challenges and improving performance on knowledge-intensive tasks.

Contribution

The paper proposes PiCA, a novel pivot-based credit assignment method that enhances reward signals by leveraging success probabilities and historical context, outperforming existing methods.

Findings

PiCA achieves 15.2% and 2.2% improvements on 3B and 7B models.

PiCA outperforms strong baselines across seven QA benchmarks.

PiCA maintains distributional consistency while providing dense, pivot-aware guidance.

Abstract

Large Language Model (LLM)-based search agents trained with reinforcement learning (RL) have significantly improved the performance of knowledge-intensive tasks. However, existing methods encounter critical challenges in long-horizon credit assignment: (i) Reward Sparsity, where models receive only outcome feedback without step-level guidance to differentiate action quality; (ii) Isolated Credit, where credit is assigned to steps independently, failing to capture sequential dependencies; and (iii) Distributional Shift, where rewards are estimated on templates that deviate from the model's natural generative distribution. To address these issues, we propose Pivot-Based Credit Assignment (PiCA), a novel step reward mechanism that reformulates the search trajectory as a sequential process of cumulative search progress. Unlike prior isolated step rewards, PiCA defines process rewards as success probabilities dependent on the historical context based on Potential-Based Reward Shaping (PBRS). This approach identifies pivot steps, which comprise target golden sub-queries and sub-answers derived from historical trajectories, as information peaks that significantly boost the likelihood of a correct final answer. By anchoring these step rewards to the final task objective, PiCA provides dense, pivot-aware and trajectory-dependent guidance while maintaining distributional consistency. Extensive experiments show that PiCA outperforms existing strong baselines across seven knowledge-intensive QA benchmarks, achieving 15.2% and 2.2% improvements for 3B and 7B models. The consistent performance gains across various models show PiCA's robust generalization. The code is available at https://github.com/novdream/PiCA.