Sparsifying Sparse Representations for Passage Retrieval by Top-$k$ Masking

TL;DR

This paper introduces a simple top-$k$ masking technique and self-learning approach to sparsify lexical representations for passage retrieval, improving efficiency while maintaining effectiveness.

Contribution

It presents a novel top-$k$ masking scheme combined with self-learning to sparsify lexical representations, building on SPLADE for better passage retrieval.

Findings

Competitive with sophisticated models

Balances effectiveness and efficiency

Simplifies lexical representation sparsification

Abstract

Sparse lexical representation learning has demonstrated much progress in improving passage retrieval effectiveness in recent models such as DeepImpact, uniCOIL, and SPLADE. This paper describes a straightforward yet effective approach for sparsifying lexical representations for passage retrieval, building on SPLADE by introducing a top-$k$ masking scheme to control sparsity and a self-learning method to coax masked representations to mimic unmasked representations. A basic implementation of our model is competitive with more sophisticated approaches and achieves a good balance between effectiveness and efficiency. The simplicity of our methods opens the door for future explorations in lexical representation learning for passage retrieval.