HotelQuEST: Balancing Quality and Efficiency in Agentic Search

TL;DR

HotelQuEST introduces a comprehensive benchmark for agentic search that evaluates both quality and efficiency, addressing real-world complexities like underspecified preferences and complex queries to improve practical deployment of LLM-based search systems.

Contribution

This work presents HotelQuEST, a new benchmark with 214 hotel search queries, and explores evaluation methods for underspecified preferences, highlighting efficiency issues in current agentic search systems.

Findings

LLM-based agents outperform traditional retrievers in accuracy.

Current systems exhibit high costs due to redundant tool calls.

Significant potential for cost-aware optimization in agentic search.

Abstract

Agentic search has emerged as a promising paradigm for adaptive retrieval systems powered by large language models (LLMs). However, existing benchmarks primarily focus on quality, overlooking efficiency factors that are critical for real-world deployment. Moreover, real-world user queries often contain underspecified preferences, a challenge that remains largely underexplored in current agentic search evaluation. As a result, many agentic search systems remain impractical despite their impressive performance. In this work, we introduce HotelQuEST, a benchmark comprising 214 hotel search queries that range from simple factual requests to complex queries, enabling evaluation across the full spectrum of query difficulty. We further address the challenge of evaluating underspecified user preferences by collecting clarifications that make annotators' implicit preferences explicit for evaluation. We find that LLM-based agents achieve higher accuracy than traditional retrievers, but at substantially higher costs due to redundant tool calls and suboptimal routing that fails to match query complexity to model capability. Our analysis exposes inefficiencies in current agentic search systems and demonstrates substantial potential for cost-aware optimization.