A Picture of Agentic Search

TL;DR

This paper highlights the shift in Information Retrieval due to increased use of automated agents, identifies the lack of agent-specific datasets, and introduces the ASQ dataset and toolkit for better evaluation of agentic search systems.

Contribution

It develops a methodology for collecting agent search data and releases the ASQ dataset and toolkit to support research on agent-centric IR evaluation.

Findings

ASQ dataset includes reasoning queries, retrieved documents, and agent thoughts.

The dataset covers diverse agents and retrieval pipelines.

Toolkit enables extension to new agents and datasets.

Abstract

With automated systems increasingly issuing search queries alongside humans, Information Retrieval (IR) faces a major shift. Yet IR remains human-centred, with systems, evaluation metrics, user models, and datasets designed around human queries and behaviours. Consequently, IR operates under assumptions that no longer hold in practice, with changes to workload volumes, predictability, and querying behaviours. This misalignment affects system performance and optimisation: caching may lose effectiveness, query pre-processing may add overhead without improving results, and standard metrics may mismeasure satisfaction. Without adaptation, retrieval models risk satisfying neither humans, nor the emerging user segment of agents. However, datasets capturing agent search behaviour are lacking, which is a critical gap given IR's historical reliance on data-driven evaluation and optimisation. We develop a methodology for collecting all the data produced and consumed by agentic retrieval-augmented systems when answering queries, and we release the Agentic Search Queryset (ASQ) dataset. ASQ contains reasoning-induced queries, retrieved documents, and thoughts for queries in HotpotQA, Researchy Questions, and MS MARCO, for 3 diverse agents and 2 retrieval pipelines. The accompanying toolkit enables ASQ to be extended to new agents, retrievers, and datasets.