Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective

TL;DR

This paper compares diffusion and autoregressive language models for text embeddings, highlighting the advantages of bidirectional architectures in capturing global context and improving retrieval and reasoning tasks.

Contribution

It introduces the first systematic study of diffusion language embedding models, demonstrating their superior performance over LLM-based embeddings in various retrieval and reasoning tasks.

Findings

Diffusion models outperform LLMs by 20% on long-document retrieval.

Diffusion models improve reasoning-intensive retrieval by 8%.

Bidirectional attention is key for encoding global context.

Abstract

Large language model (LLM)-based embedding models, benefiting from large scale pre-training and post-training, have begun to surpass BERT and T5-based models on general-purpose text embedding tasks such as document retrieval. However, a fundamental limitation of LLM embeddings lies in the unidirectional attention used during autoregressive pre-training, which misaligns with the bidirectional nature of text embedding tasks. To this end, We propose adopting diffusion language models for text embeddings, motivated by their inherent bidirectional architecture and recent success in matching or surpassing LLMs especially on reasoning tasks. We present the first systematic study of the diffusion language embedding model, which outperforms the LLM-based embedding model by 20% on long-document retrieval, 8% on reasoning-intensive retrieval, 2% on instruction-following retrieval, and achieve competitive performance on traditional text embedding benchmarks. Our analysis verifies that bidirectional attention is crucial for encoding global context in long and complex text.