RouterRetriever: Routing over a Mixture of Expert Embedding Models

TL;DR

RouterRetriever introduces a routing-based retrieval model that leverages domain-specific experts, outperforming traditional single-model approaches on diverse datasets by selecting the most relevant expert for each query.

Contribution

This work pioneers the use of routing over a mixture of domain-specific expert embedding models in retrieval, demonstrating improved performance and flexibility.

Findings

Outperforms MSMARCO-trained models by +2.1 nDCG@10

Surpasses multi-task models by +3.2 nDCG@10

Routing mechanism outperforms other techniques by +1.8 on average

Abstract

Information retrieval methods often rely on a single embedding model trained on large, general-domain datasets like MSMARCO. While this approach can produce a retriever with reasonable overall performance, they often underperform models trained on domain-specific data when testing on their respective domains. Prior work in information retrieval has tackled this through multi-task training, but the idea of routing over a mixture of domain-specific expert retrievers remains unexplored despite the popularity of such ideas in language model generation research. In this work, we introduce RouterRetriever, a retrieval model that leverages a mixture of domain-specific experts by using a routing mechanism to select the most appropriate expert for each query. RouterRetriever is lightweight and allows easy addition or removal of experts without additional training. Evaluation on the BEIR benchmark demonstrates that RouterRetriever outperforms both models trained on MSMARCO (+2.1 absolute nDCG@10) and multi-task models (+3.2). This is achieved by employing our routing mechanism, which surpasses other routing techniques (+1.8 on average) commonly used in language modeling. Furthermore, the benefit generalizes well to other datasets, even in the absence of a specific expert on the dataset. RouterRetriever is the first work to demonstrate the advantages of routing over a mixture of domain-specific expert embedding models as an alternative to a single, general-purpose embedding model, especially when retrieving from diverse, specialized domains.