DIFFA: Large Language Diffusion Models Can Listen and Understand

TL;DR

DIFFA introduces a diffusion-based large audio-language model that effectively understands spoken language, demonstrating competitive performance with limited training data and opening new avenues for speech-driven AI.

Contribution

This work is the first to apply diffusion models to large-scale audio-language understanding, integrating a frozen diffusion model with a dual-adapter architecture and a two-stage training pipeline.

Findings

DIFFA outperforms autoregressive baselines on major benchmarks.

Effective with only 960 hours of ASR data and 127 hours of synthetic instruction data.

Demonstrates potential of diffusion models for scalable audio understanding.

Abstract

Recent advances in large language models (LLMs) have shown remarkable capabilities across textual and multimodal domains. In parallel, diffusion-based language models have emerged as a promising alternative to the autoregressive paradigm, offering improved controllability, bidirectional context modeling, and robust generation. However, their application to the audio modality remains underexplored. In this work, we introduce \textbf{DIFFA}, the first diffusion-based large audio-language model designed to perform spoken language understanding. DIFFA integrates a frozen diffusion language model with a lightweight dual-adapter architecture that bridges speech understanding and natural language reasoning. We employ a two-stage training pipeline: first, aligning semantic representations via an ASR objective; then, learning instruction-following abilities through synthetic audio-caption pairs automatically generated by prompting LLMs. Despite being trained on only 960 hours of ASR and 127 hours of synthetic instruction data, DIFFA demonstrates competitive performance on major benchmarks, including MMSU, MMAU, and VoiceBench, outperforming several autoregressive open-source baselines. Our results reveal the potential of diffusion-based language models for efficient and scalable audio understanding, opening a new direction for speech-driven AI. Our code will be available at https://github.com/NKU-HLT/DIFFA.git.