LTRR: Learning To Rank Retrievers for LLMs

TL;DR

LTRR introduces a learning-to-rank framework for dynamically selecting retrievers in RAG systems, significantly improving performance and generalization across diverse query types.

Contribution

The paper presents LTRR, a novel query routing method that learns to rank retrievers for RAG, outperforming fixed retrievers and enhancing adaptability.

Findings

Routing-based RAG outperforms single-retriever baselines.

Training with the AC objective and pairwise ranking yields best results.

LTRR generalizes better to out-of-distribution queries.

Abstract

Retrieval-Augmented Generation (RAG) systems typically rely on a single fixed retriever, despite growing evidence that no single retriever performs optimally across all query types. In this paper, we explore a query routing approach that dynamically selects from a pool of retrievers based on the query, using both train-free heuristics and learned routing models. We frame routing as a learning-to-rank problem and introduce LTRR, a framework that Learns To Rank Retrievers according to their expected contribution to downstream RAG performance. Through experiments on diverse question-answering benchmarks with controlled variations in query types, we demonstrate that routing-based RAG consistently surpasses the strongest single-retriever baselines. The gains are particularly substantial when training with the Answer Correctness (AC) objective and when using pairwise ranking methods, with XGBoost yielding the best results. Additionally, our approach exhibits stronger generalization to out-of-distribution queries. Overall, our results underscore the critical role of both training strategy and optimization metric choice in effective query routing for RAG systems.