CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrieval

TL;DR

This paper introduces CoT-MoTE, a novel pre-training method for passage retrieval that uses mixture-of-experts to better encode queries and passages, resulting in improved retrieval accuracy and balanced embeddings.

Contribution

It proposes a new pre-training approach combining textual-specific experts with shared attention for dual-encoder passage retrieval models.

Findings

Improved retrieval performance on large-scale benchmarks.

More balanced discrimination of embedding spaces.

Effective encoding of query and passage properties.

Abstract

Passage retrieval aims to retrieve relevant passages from large collections of the open-domain corpus. Contextual Masked Auto-Encoding has been proven effective in representation bottleneck pre-training of a monolithic dual-encoder for passage retrieval. Siamese or fully separated dual-encoders are often adopted as basic retrieval architecture in the pre-training and fine-tuning stages for encoding queries and passages into their latent embedding spaces. However, simply sharing or separating the parameters of the dual-encoder results in an imbalanced discrimination of the embedding spaces. In this work, we propose to pre-train Contextual Masked Auto-Encoder with Mixture-of-Textual-Experts (CoT-MoTE). Specifically, we incorporate textual-specific experts for individually encoding the distinct properties of queries and passages. Meanwhile, a shared self-attention layer is still kept for unified attention modeling. Results on large-scale passage retrieval benchmarks show steady improvement in retrieval performances. The quantitive analysis also shows a more balanced discrimination of the latent embedding spaces.