A Study on the Efficiency and Generalization of Light Hybrid Retrievers

TL;DR

This paper introduces LITE, a memory-efficient hybrid retriever that combines sparse and dense methods, achieving high performance with significantly reduced memory usage and improved out-of-domain generalization.

Contribution

We propose LITE, a novel lightweight dense retriever trained with contrastive learning and knowledge distillation, enhancing hybrid retrieval efficiency and generalization.

Findings

LITE reduces memory by 13X compared to traditional hybrid retrievers.

Hybrid-LITE maintains 98% of BM25-DPR performance.

Light hybrid retrievers outperform individual sparse or dense retrievers on out-of-domain and adversarial datasets.

Abstract

Hybrid retrievers can take advantage of both sparse and dense retrievers. Previous hybrid retrievers leverage indexing-heavy dense retrievers. In this work, we study "Is it possible to reduce the indexing memory of hybrid retrievers without sacrificing performance"? Driven by this question, we leverage an indexing-efficient dense retriever (i.e. DrBoost) and introduce a LITE retriever that further reduces the memory of DrBoost. LITE is jointly trained on contrastive learning and knowledge distillation from DrBoost. Then, we integrate BM25, a sparse retriever, with either LITE or DrBoost to form light hybrid retrievers. Our Hybrid-LITE retriever saves 13X memory while maintaining 98.0% performance of the hybrid retriever of BM25 and DPR. In addition, we study the generalization capacity of our light hybrid retrievers on out-of-domain dataset and a set of adversarial attacks datasets. Experiments showcase that light hybrid retrievers achieve better generalization performance than individual sparse and dense retrievers. Nevertheless, our analysis shows that there is a large room to improve the robustness of retrievers, suggesting a new research direction.