DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers

TL;DR

DRAMA is a training framework that uses large language models to enhance smaller dense retrievers, improving their generalization, multilingual, and long-context capabilities while maintaining efficiency.

Contribution

It introduces a novel single-stage contrastive learning method leveraging pruned LLMs and diverse augmented data to train smaller, more effective dense retrievers.

Findings

DRAMA outperforms traditional retrievers in multilingual tasks.

It enhances long-context understanding in dense retrieval.

The framework achieves strong results across multiple languages and tasks.

Abstract

Large language models (LLMs) have demonstrated strong effectiveness and robustness while fine-tuned as dense retrievers. However, their large parameter size brings significant inference time computational challenges, including high encoding costs for large-scale corpora and increased query latency, limiting their practical deployment. While smaller retrievers offer better efficiency, they often fail to generalize effectively with limited supervised fine-tuning data. In this work, we introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers. In particular, we adopt pruned LLMs as the backbone and train on diverse LLM-augmented data in a single-stage contrastive learning setup. Experiments show that DRAMA offers better multilingual and long-context capabilities than traditional encoder-based retrievers, and achieves strong performance across multiple tasks and languages. These highlight the potential of connecting the training of smaller retrievers with the growing advancements in LLMs, bridging the gap between efficiency and generalization.