DeeperImpact: Optimizing Sparse Learned Index Structures

TL;DR

DeeperImpact enhances sparse learned index retrieval by optimizing document expansion and training strategies, significantly narrowing the effectiveness gap with dense retrievers through the use of Llama 2 and advanced training techniques.

Contribution

The paper introduces a refined DeepImpact approach using Llama 2 for query prediction and effective training strategies, improving retrieval effectiveness over previous sparse models.

Findings

Replacing T5 with Llama 2 improves retrieval quality.

Hard negatives, distillation, and CoCondenser initialization boost performance.

DeeperImpact narrows the effectiveness gap with SPLADE.

Abstract

A lot of recent work has focused on sparse learned indexes that use deep neural architectures to significantly improve retrieval quality while keeping the efficiency benefits of the inverted index. While such sparse learned structures achieve effectiveness far beyond those of traditional inverted index-based rankers, there is still a gap in effectiveness to the best dense retrievers, or even to sparse methods that leverage more expensive optimizations such as query expansion and query term weighting. We focus on narrowing this gap by revisiting and optimizing DeepImpact, a sparse retrieval approach that uses DocT5Query for document expansion followed by a BERT language model to learn impact scores for document terms. We first reinvestigate the expansion process and find that the recently proposed Doc2Query -- query filtration does not enhance retrieval quality when used with DeepImpact. Instead, substituting T5 with a fine-tuned Llama 2 model for query prediction results in a considerable improvement. Subsequently, we study training strategies that have proven effective for other models, in particular the use of hard negatives, distillation, and pre-trained CoCondenser model initialization. Our results substantially narrow the effectiveness gap with the most effective versions of SPLADE.