Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction

TL;DR

This paper introduces a novel contrastive span prediction pre-training task for text encoders to improve dense retrieval performance, avoiding decoder bypass effects and enhancing discriminative representations.

Contribution

It proposes a new contrastive span prediction method that pre-trains encoders independently, leading to more discriminative text representations for dense retrieval tasks.

Findings

Outperforms existing pre-training methods on retrieval benchmarks.

Effectively learns discriminative representations without decoder bypass.

Significantly improves dense retrieval accuracy.

Abstract

Dense retrieval has shown promising results in many information retrieval (IR) related tasks, whose foundation is high-quality text representation learning for effective search. Some recent studies have shown that autoencoder-based language models are able to boost the dense retrieval performance using a weak decoder. However, we argue that 1) it is not discriminative to decode all the input texts and, 2) even a weak decoder has the bypass effect on the encoder. Therefore, in this work, we introduce a novel contrastive span prediction task to pre-train the encoder alone, but still retain the bottleneck ability of the autoencoder. % Therefore, in this work, we propose to drop out the decoder and introduce a novel contrastive span prediction task to pre-train the encoder alone. The key idea is to force the encoder to generate the text representation close to its own random spans while far away from others using a group-wise contrastive loss. In this way, we can 1) learn discriminative text representations efficiently with the group-wise contrastive learning over spans and, 2) avoid the bypass effect of the decoder thoroughly. Comprehensive experiments over publicly available retrieval benchmark datasets show that our approach can outperform existing pre-training methods for dense retrieval significantly.