m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder

TL;DR

m3BERT is a versatile, multi-lingual embedding model with a novel pretraining strategy that allows adaptation to various resource and accuracy constraints, outperforming existing models in industrial retrieval tasks.

Contribution

The paper introduces m3BERT, a multi-lingual, multi-granular transformer model with a new pretraining approach that enhances adaptability and performance in resource-constrained industrial retrieval systems.

Findings

m3BERT outperforms state-of-the-art models on Bing-Click dataset.

The model effectively adapts to diverse resource and accuracy requirements.

Multigranular pretraining improves generalization on public datasets.

Abstract

Embedding models are pivotal in industrial information retrieval systems like search and advertising. However, existing pretrained models often exhibit fixed architectures and embedding dimensionalities, posing significant challenges when adapting them to diverse deployment scenarios with varying business-driven constraints. A common practice involves fine-tuning with partial parameter initialization from larger pretrained models for resource-constrained tasks. This method is often suboptimal as the misalignment between pretraining and downstream usage prevents full realization of pretraining benefits. To address this limitation, we introduce m3BERT: a Modern, Multi-lingual, Matryoshka Bidirectional Encoder, which features a novel pretraining strategy that jointly optimizes representations across both transformer layers and multiple embedding dimensions. This enables a single model to be tailored to varied resource and accuracy targets while maintaining consistency with pretraining. Incorporating recent architectural improvements, m3BERT uses a three-stage pretraining: monolingual pretraining, multilingual adaptation to serve diverse user bases, and crucial continual pretraining on a massive web domain corpus to enhance utility in commercial retrieval. m3BERT significantly outperforms state-of-the-art embedding models in Bing-Click, a large-scale industrial retrieval dataset, showcasing its practical versatility as an efficient foundation for resource-aware industrial retrieval systems. Further experiments on public datasets also confirm the general effectiveness of our multigranular Matryoshka pretraining strategy.