SGR-Bench: Benchmarking Search Agents on State-Gated Retrieval

TL;DR

SGR-Bench is a new benchmark designed to evaluate search agents on their ability to perform state-gated retrieval tasks, which involve establishing correct site-specific retrieval states before producing answers.

Contribution

The paper introduces SGR-Bench, a comprehensive benchmark for evaluating agents on specialized retrieval tasks requiring state establishment, with detailed analysis of system performance and failure modes.

Findings

The strongest system achieves only 66.18% item-level F1 on SGR-Bench.

Agents often reach relevant sources but establish incorrect retrieval states.

Retrieval-scope drift and criterion mismatch are major failure causes.

Abstract

Recent advances in large language models and tool-using agents have expanded the range of benchmarked web tasks. Yet an important class of specialized retrieval tasks remains undercharacterized. On many specialized data-retrieval websites, answer-bearing evidence becomes accessible only after establishing the correct site-specific retrieval state through filters, views, hierarchies, or scopes. We term this capability state-gated retrieval (SGR). We introduce SGR-Bench, a benchmark for this setting containing 100 expert-curated tasks spanning six source families and 12 public data ecosystems. Each task requires discovering the appropriate website and configuring its site-specific retrieval state to produce a structured answer. SGR-Bench pairs constraint-guided and goal-oriented formulations of the same underlying problems, enabling controlled comparisons between explicit and implicit guidance for state-gated retrieval. We evaluate eight CLI-based agentic LLM systems and three commercial search-agent products. On SGR-Bench, the strongest system reaches only 66.18% item-level F1, while row-level F1 remains much lower. A manual audit of 156 analyzable failed CLI trajectories shows why: agents often reach a relevant web source, but establish the wrong site-specific retrieval state. Retrieval-scope drift (37.2%) and criterion mismatch (27.6%) dominate, whereas final answer composition accounts for only 10.3%. The dataset and single-case evaluation instructions are available at https://huggingface.co/datasets/PKUAIWeb/SGR-BENCH.