LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services

TL;DR

This paper introduces LocalSearchBench, a comprehensive benchmark for agentic search in local life services, highlighting the challenges faced by current models in multi-hop reasoning and domain-specific tasks.

Contribution

It presents the first large-scale benchmark dataset and environment for agentic search in local life services, addressing a gap in domain-specific multi-step reasoning evaluation.

Findings

State-of-the-art LRMs perform poorly on LocalSearchBench

Models struggle with completeness and faithfulness in local service queries

Highlighting the need for domain-specific training and benchmarks

Abstract

Recent advances in large reasoning models LRMs have enabled agentic search systems to perform complex multi-step reasoning across multiple sources. However, most studies focus on general information retrieval and rarely explores vertical domains with unique challenges. In this work, we focus on local life services and introduce LocalSearchBench, which encompass diverse and complex business scenarios. Real-world queries in this domain are often ambiguous and require multi-hop reasoning across merchants and products, remaining challenging and not fully addressed. As the first comprehensive benchmark for agentic search in local life services, LocalSearchBench comprises a database of over 1.3M merchant entries across 6 service categories and 9 major cities, and 900 multi-hop QA tasks from real user queries that require multi-step reasoning. We also developed LocalPlayground, a unified environment integrating multiple tools for LRMs interaction. Experiments show that even state-of-the-art LRMs struggle on LocalSearchBench: the best model (DeepSeek-V3.2) achieves only 35.60% correctness, and most models have issues with completeness (average 60.32%) and faithfulness (average 30.72%). This highlights the need for specialized benchmarks and domain-specific agent training in local life services. Code, Benchmark, and Leaderboard are available at https://localsearchbench.github.io/.