Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space

TL;DR

This paper systematically evaluates diffusion and autoregressive vision language models in multimodal embedding tasks, revealing diffusion models generally underperform due to alignment issues, with LaViDa being the closest competitor.

Contribution

First systematic study of converting Multimodal diffusion LLMs into embedding models, comparing their performance to autoregressive VLMs across multiple tasks.

Findings

Diffusion models underperform autoregressive models in embedding tasks.

LaViDa performs close to autoregressive models with small performance gaps.

Image-text alignment issues limit diffusion models' embedding effectiveness.

Abstract

Embedding models are a fundamental component of modern AI systems such as semantic search and retrieval-augmented generation. Recent advances in large foundation models have substantially accelerated the development of embedding models, including those based on Large Language Models (LLMs), Vision Language Models (VLMs), and Multimodal LLMs. More recently, Large Diffusion Language Models (dLLMs) and Multimodal dLLMs have emerged as competitive alternatives to autoregressive models, offering advantages such as bidirectional attention and parallel generation. This progress naturally raises a critical yet unexplored question: can Multimodal dLLMs serve as effective multimodal embedding models? To answer this, we present the first systematic study of converting Multimodal dLLMs into embedding models. We evaluate state-of-the-art Multimodal dLLMs and Autoregressive VLMs across three categories of embedding tasks: classification, visual question answering, and information retrieval. Our results show that Multimodal dLLM embeddings generally underperform their autoregressive VLM counterparts. The stronger diffusion-based model, LaViDa, lags by only 3.5 points on classification, 2.5 points on VQA, and 4.4 points on retrieval tasks, whereas the other diffusion-based model, MMaDA, exhibits substantially larger performance gaps, exceeding 20 points across all tasks. Further analysis reveals insufficient image-text alignment in diffusion-based models, accounting for the observed limitations in their embedding performance.