A Picture of Agentic Search
Francesca Pezzuti, Ophir Frieder, Fabrizio Silvestri, Sean MacAvaney, Nicola Tonellotto

TL;DR
This paper highlights the shift in Information Retrieval due to increased use of automated agents, identifies the lack of agent-specific datasets, and introduces the ASQ dataset and toolkit for better evaluation of agentic search systems.
Contribution
It develops a methodology for collecting agent search data and releases the ASQ dataset and toolkit to support research on agent-centric IR evaluation.
Findings
ASQ dataset includes reasoning queries, retrieved documents, and agent thoughts.
The dataset covers diverse agents and retrieval pipelines.
Toolkit enables extension to new agents and datasets.
Abstract
With automated systems increasingly issuing search queries alongside humans, Information Retrieval (IR) faces a major shift. Yet IR remains human-centred, with systems, evaluation metrics, user models, and datasets designed around human queries and behaviours. Consequently, IR operates under assumptions that no longer hold in practice, with changes to workload volumes, predictability, and querying behaviours. This misalignment affects system performance and optimisation: caching may lose effectiveness, query pre-processing may add overhead without improving results, and standard metrics may mismeasure satisfaction. Without adaptation, retrieval models risk satisfying neither humans, nor the emerging user segment of agents. However, datasets capturing agent search behaviour are lacking, which is a critical gap given IR's historical reliance on data-driven evaluation and optimisation. We…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInformation Retrieval and Search Behavior · Personal Information Management and User Behavior · Web Data Mining and Analysis
