Align then Train: Efficient Retrieval Adapter Learning

TL;DR

The paper introduces ERA, a two-stage training framework that aligns large and lightweight embedding models for efficient retrieval of complex queries, reducing the need for extensive fine-tuning.

Contribution

ERA offers a label-efficient, two-stage approach to improve retrieval performance by aligning embedding spaces and adapting representations without re-indexing the corpus.

Findings

ERA improves retrieval in low-label settings.

Outperforms methods requiring more labeled data.

Effectively combines strong query and weak document embedders.

Abstract

Dense retrieval systems increasingly need to handle complex queries. In many realistic settings, users express intent through long instructions or task-specific descriptions, while target documents remain relatively simple and static. This asymmetry creates a retrieval mismatch: understanding queries may require strong reasoning and instruction-following, whereas efficient document indexing favors lightweight encoders. Existing retrieval systems often address this mismatch by directly improving the embedding model, but fine-tuning large embedding models to better follow such instructions is computationally expensive, memory-intensive, and operationally burdensome. To address this challenge, we propose Efficient Retrieval Adapter (ERA), a label-efficient framework that trains retrieval adapters in two stages: self-supervised alignment and supervised adaptation. Inspired by the pre-training and supervised fine-tuning stages of LLMs, ERA first aligns the embedding spaces of a large query embedder and a lightweight document embedder, and then uses limited labeled data to adapt the query-side representation, bridging both the representation gap between embedding models and the semantic gap between complex queries and simple documents without re-indexing the corpus. Experiments on the MAIR benchmark, spanning 126 retrieval tasks across 6 domains, show that ERA improves retrieval in low-label settings, outperforms methods that rely on larger amounts of labeled data, and effectively combines stronger query embedders with weaker document embedders across domains.