Diffusion-Pretrained Dense and Contextual Embeddings

TL;DR

This paper introduces pplx-embed, a family of multilingual embedding models using diffusion-based pretraining and contrastive learning, achieving state-of-the-art retrieval performance on multiple benchmarks and real-world web-scale data.

Contribution

It presents diffusion-pretrained multilingual embeddings with bidirectional context capture, novel pooling strategies, and two model variants optimized for retrieval tasks.

Findings

pplx-embed-v1 achieves competitive results on multiple benchmarks.

pplx-embed-context-v1 sets new records on ConTEB.

Models perform well in large-scale, real-world search scenarios.

Abstract

In this report, we introduce pplx-embed, a family of multilingual embedding models that employ multi-stage contrastive learning on a diffusion-pretrained language model backbone for web-scale retrieval. By leveraging bidirectional attention through diffusion-based pretraining, our models capture comprehensive bidirectional context within passages, enabling the use of mean pooling and a late chunking strategy to better preserve global context across long documents. We release two model types: pplx-embed-v1 for standard retrieval, and pplx-embed-context-v1 for contextualized embeddings that incorporate global document context into passage representations. pplx-embed-v1 achieves competitive performance on the MTEB(Multilingual, v2), MTEB(Code), MIRACL, BERGEN, and ToolRet retrieval benchmarks, while pplx-embed-context-v1 sets new records on the ConTEB benchmark. Beyond public benchmarks, pplx-embed-v1 demonstrates strong performance on our internal evaluation suite, focusing on real-world, large-scale search scenarios constructed from 1B production web pages. These results validate the models' effectiveness in production environments where retrieval quality and efficiency are critical at scale.