Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents

TL;DR

This paper investigates how to optimize reasoning and reranking strategies in deep search agents by introducing an effective token cost metric, demonstrating reranking's efficiency in improving accuracy at lower costs.

Contribution

It introduces the effective token cost metric and analyzes reranking tradeoffs in deep search pipelines, providing insights into optimizing reasoning budgets.

Findings

Reranking improves retrieval and end-to-end accuracy.

Moderate reranking often outperforms increasing reasoning effort.

Reranking achieves comparable accuracy at lower token costs.

Abstract

Deep research agents rely on iterative retrieval and reasoning to answer complex queries, but scaling test-time computation raises significant efficiency concerns. We study how to allocate reasoning budget in deep search pipelines, focusing on the role of listwise reranking. Using the BrowseComp-Plus benchmark, we analyze tradeoffs between model scale, reasoning effort, reranking depth, and total token cost via a novel effective token cost (ETC) metric. Our results show that reranking consistently improves retrieval and end-to-end accuracy, and that moderate reranking often yields larger gains than increasing search-time reasoning, achieving comparable accuracy at substantially lower cost. All our code is available at https://github.com/texttron/BrowseComp-Plus.git