In Defense of Cross-Encoders for Zero-Shot Retrieval

TL;DR

This paper demonstrates that large cross-encoders significantly outperform bi-encoders in zero-shot retrieval tasks, especially in out-of-domain scenarios, due to their greater parameter count and interaction complexity.

Contribution

It provides a comprehensive analysis of how model size and architecture influence zero-shot retrieval performance, highlighting the advantages of cross-encoders over bi-encoders.

Findings

Cross-encoders outperform bi-encoders of similar size in multiple tasks.

Increasing model size yields larger gains in out-of-domain retrieval.

Using bi-encoders as first-stage retrievers offers no advantage over BM25 in out-of-domain settings.

Abstract

Bi-encoders and cross-encoders are widely used in many state-of-the-art retrieval pipelines. In this work we study the generalization ability of these two types of architectures on a wide range of parameter count on both in-domain and out-of-domain scenarios. We find that the number of parameters and early query-document interactions of cross-encoders play a significant role in the generalization ability of retrieval models. Our experiments show that increasing model size results in marginal gains on in-domain test sets, but much larger gains in new domains never seen during fine-tuning. Furthermore, we show that cross-encoders largely outperform bi-encoders of similar size in several tasks. In the BEIR benchmark, our largest cross-encoder surpasses a state-of-the-art bi-encoder by more than 4 average points. Finally, we show that using bi-encoders as first-stage retrievers provides no gains in comparison to a simpler retriever such as BM25 on out-of-domain tasks. The code is available at https://github.com/guilhermemr04/scaling-zero-shot-retrieval.git