Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search

TL;DR

This study systematically evaluates how search depth, retrieval methods, and budget constraints impact the accuracy and cost of agentic LLM search systems, providing practical configuration guidance.

Contribution

It introduces BCAS, a model-agnostic evaluation framework, and offers empirical insights into optimizing budgeted agentic retrieval for improved accuracy.

Findings

Accuracy improves with search depth up to a small cap.

Hybrid retrieval with lightweight re-ranking yields significant gains.

Larger completion budgets benefit synthesis tasks like HotpotQA.

Abstract

Agentic Retrieval-Augmented Generation (RAG) systems combine iterative search, planning prompts, and retrieval backends, but deployed settings impose explicit budgets on tool calls and completion tokens. We present a controlled measurement study of how search depth, retrieval strategy, and completion budget affect accuracy and cost under fixed constraints. Using Budget-Constrained Agentic Search (BCAS), a model-agnostic evaluation harness that surfaces remaining budget and gates tool use, we run comparisons across six LLMs and three question-answering benchmarks. Across models and datasets, accuracy improves with additional searches up to a small cap, hybrid lexical and dense retrieval with lightweight re-ranking produces the largest average gains in our ablation grid, and larger completion budgets are most helpful on HotpotQA-style synthesis. These results provide practical guidance for configuring budgeted agentic retrieval pipelines and are accompanied by reproducible prompts and evaluation settings.