Repurposing Synthetic Data for Fine-grained Search Agent Supervision

TL;DR

This paper introduces E-GRPO, a novel training framework that leverages entity information in synthetic data to improve the learning and reasoning efficiency of search agents in complex QA tasks.

Contribution

The paper proposes E-GRPO, a new entity-aware reward method that enhances training by utilizing discarded entity information, leading to better performance and reasoning efficiency.

Findings

E-GRPO outperforms the baseline GRPO in diverse QA benchmarks.

E-GRPO enables learning from near-miss samples through partial rewards.

Models trained with E-GRPO require fewer tool calls, indicating more efficient reasoning.

Abstract

LLM-based search agents are increasingly trained on entity-centric synthetic data to solve complex, knowledge-intensive tasks. However, prevailing training methods like Group Relative Policy Optimization (GRPO) discard this rich entity information, relying instead on sparse, outcome-based rewards. This critical limitation renders them unable to distinguish informative "near-miss" samples-those with substantially correct reasoning but a flawed final answer-from complete failures, thus discarding valuable learning signals. We address this by leveraging the very entities discarded during training. Our empirical analysis reveals a strong positive correlation between the number of ground-truth entities identified during an agent's reasoning process and final answer accuracy. Building on this insight, we introduce Entity-aware Group Relative Policy Optimization (E-GRPO), a novel framework that formulates a dense entity-aware reward function. E-GRPO assigns partial rewards to incorrect samples proportional to their entity match rate, enabling the model to effectively learn from these "near-misses". Experiments on diverse question-answering (QA) and deep research benchmarks show that E-GRPO consistently and significantly outperforms the GRPO baseline. Furthermore, our analysis reveals that E-GRPO not only achieves superior accuracy but also induces more efficient reasoning policies that require fewer tool calls, demonstrating a more effective and sample-efficient approach to aligning search agents.