RetroMAE v2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models

TL;DR

RetroMAE v2 introduces a duplex masked auto-encoder that jointly pre-trains [CLS] and ordinary token embeddings, significantly enhancing retrieval-oriented language models' semantic representation and transferability for tasks like web search and QA.

Contribution

It proposes a novel duplex auto-encoding pre-training method that improves the semantic capacity of all token embeddings, not just [CLS], for retrieval tasks.

Findings

Achieves remarkable improvements on MS MARCO and BEIR benchmarks.

Enhances the transferability of retrieval-oriented language models.

Provides a simple yet effective pre-training approach.

Abstract

To better support retrieval applications such as web search and question answering, growing effort is made to develop retrieval-oriented language models. Most of the existing works focus on improving the semantic representation capability for the contextualized embedding of [CLS] token. However, recent study shows that the ordinary tokens besides [CLS] may provide extra information, which helps to produce a better representation effect. As such, it's necessary to extend the current methods where all contextualized embeddings can be jointly pre-trained for the retrieval tasks. With this motivation, we propose a new pre-training method: duplex masked auto-encoder, a.k.a. DupMAE, which targets on improving the semantic representation capacity for the contextualized embeddings of both [CLS] and ordinary tokens. It introduces two decoding tasks: one is to reconstruct the original input sentence based on the [CLS] embedding, the other one is to minimize the bag-of-words loss (BoW) about the input sentence based on the entire ordinary tokens' embeddings. The two decoding losses are added up to train a unified encoding model. The embeddings from [CLS] and ordinary tokens, after dimension reduction and aggregation, are concatenated as one unified semantic representation for the input. DupMAE is simple but empirically competitive: with a small decoding cost, it substantially contributes to the model's representation capability and transferability, where remarkable improvements are achieved on MS MARCO and BEIR benchmarks.