Are LLM-Based Retrievers Worth Their Cost? An Empirical Study of Efficiency, Robustness, and Reasoning Overhead

TL;DR

This study evaluates the efficiency, robustness, and reasoning overhead of LLM-based retrievers across multiple tasks, highlighting trade-offs between effectiveness and latency, and assessing confidence calibration issues.

Contribution

It provides a comprehensive empirical analysis of various retrievers, extending evaluation metrics, and quantifies reasoning overhead and confidence calibration challenges.

Findings

Some reasoning-specialized retrievers achieve high effectiveness with competitive throughput.

Large LLM-based bi-encoders often incur high latency with modest gains.

Confidence scores are unreliable for downstream decision-making.

Abstract

Large language model retrievers improve performance on complex queries, but their practical value depends on efficiency, robustness, and reliable confidence signals in addition to accuracy. We reproduce a reasoning-intensive retrieval benchmark (BRIGHT) across 12 tasks and 14 retrievers, and extend evaluation with cold-start indexing cost, query latency distributions and throughput, corpus scaling, robustness to controlled query perturbations, and confidence use (AUROC) for predicting query success. We also quantify \emph{reasoning overhead} by comparing standard queries to five provided reasoning-augmented variants, measuring accuracy gains relative to added latency. We find that some reasoning-specialized retrievers achieve strong effectiveness while remaining competitive in throughput, whereas several large LLM-based bi-encoders incur substantial latency for modest gains. Reasoning augmentation incurs minimal latency for sub-1B encoders but exhibits diminishing returns for top retrievers and may reduce performance on formal math/code domains. Confidence calibration is consistently weak across model families, indicating that raw retrieval scores are unreliable for downstream routing without additional calibration. We release all code and artifacts for reproducibility.