LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

TL;DR

LLaDA-V introduces a diffusion-based multimodal large language model that effectively integrates visual instruction tuning, achieving state-of-the-art performance in multimodal understanding despite weaker textual-only performance.

Contribution

This work presents LLaDA-V, a novel diffusion-based multimodal large language model that combines visual instruction tuning with masked diffusion models, diverging from autoregressive approaches.

Findings

LLaDA-V performs well on multimodal tasks despite weaker textual performance.

LLaDA-V is competitive with LLaMA3-V and narrows the gap with Qwen2-VL.

It achieves state-of-the-art results in multimodal understanding.

Abstract

In this work, we introduce LLaDA-V, a purely diffusion-based Multimodal Large Language Model (MLLM) that integrates visual instruction tuning with masked diffusion models, representing a departure from the autoregressive paradigms dominant in current multimodal approaches. Built upon LLaDA, a representative large language diffusion model, LLaDA-V incorporates a vision encoder and MLP connector that projects visual features into the language embedding space, enabling effective multimodal alignment. Our empirical investigation reveals several intriguing results: First, LLaDA-V demonstrates promising multimodal performance despite its language model being weaker on purely textual tasks than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same instruction data, LLaDA-V is highly competitive to LLaMA3-V across multimodal tasks with better data scalability. It also narrows the performance gap to Qwen2-VL, suggesting the effectiveness of its architecture for multimodal tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal understanding compared to existing hybrid autoregressive-diffusion and purely diffusion-based MLLMs. Our findings suggest that large language diffusion models show promise in multimodal contexts and warrant further investigation in future research. Project page and codes: https://ml-gsai.github.io/LLaDA-V-demo/.