LSTM-based Selective Dense Text Retrieval Guided by Sparse Lexical Retrieval

TL;DR

This paper introduces CluSD, a fast and memory-efficient method that combines dense and sparse retrieval techniques using clustering and LSTM guidance to improve search efficiency on large datasets.

Contribution

It proposes a novel cluster-based selective dense retrieval approach guided by sparse lexical retrieval, enhancing speed and reducing memory overhead.

Findings

CluSD outperforms baseline methods in search speed and accuracy.

It effectively reduces memory usage during retrieval.

CluSD demonstrates strong performance on MS MARCO and BEIR datasets.

Abstract

This paper studies fast fusion of dense retrieval and sparse lexical retrieval, and proposes a cluster-based selective dense retrieval method called CluSD guided by sparse lexical retrieval. CluSD takes a lightweight cluster-based approach and exploits the overlap of sparse retrieval results and embedding clusters in a two-stage selection process with an LSTM model to quickly identify relevant clusters while incurring limited extra memory space overhead. CluSD triggers partial dense retrieval and performs cluster-based block disk I/O if needed. This paper evaluates CluSD and compares it with several baselines for searching in-memory and on-disk MS MARCO and BEIR datasets.