RARe: Retrieval Augmented Retrieval with In-Context Examples

TL;DR

This paper introduces RARe, a method that enhances encoder-only text retrieval models by finetuning with semantically similar in-context query-document examples, leading to improved retrieval performance and better out-of-domain generalization.

Contribution

RARe is the first approach to incorporate in-context examples into encoder-only retrieval models, demonstrating performance improvements and insights into in-context example design.

Findings

Up to +2.72% nDCG improvement on retrieval datasets

RARe shows stronger out-of-domain generalization

Analysis of in-context example augmentation strategies

Abstract

While in-context learning is well-studied with decoder-only language models (LLMs), its utility for encoder-only models remains underexplored. We study in-context learning for encoder-only models for text retrieval tasks. Can incorporating in-context examples (query-document pairs) to the target query enhance retriever performance? Our approach, RARe, finetunes a pre-trained model with in-context examples whose query is semantically similar to the target query. This approach achieves performance gains of up to +2.72% nDCG across open-domain retrieval datasets (BeIR, RAR-b) compared to using the target query only as an input. In particular, we find RARe exhibits stronger out-of-domain generalization compared to models using queries without in-context examples, similar to what is seen for in-context learning in LLMs. We further provide analysis on the design choices of in-context example augmentation for retrievers and lay the foundation for future work.