DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding

TL;DR

DIFFA-2 introduces a practical diffusion-based large language model for general audio understanding, demonstrating improved performance and efficiency over previous models through a comprehensive training curriculum and open-source implementation.

Contribution

It presents DIFFA-2, a diffusion-based LALM that enhances audio understanding with a novel training pipeline and architectural upgrades, making diffusion models viable for large-scale audio tasks.

Findings

DIFFA-2 outperforms DIFFA in multiple benchmarks.

DIFFA-2 is competitive with autoregressive models under practical budgets.

The model is trained solely on open-source data.

Abstract

Autoregressive (AR) large audio language models (LALMs) such as Qwen-2.5-Omni have achieved strong performance on audio understanding and interaction, but scaling them remains costly in data and computation, and strictly sequential decoding limits inference efficiency. Diffusion large language models (dLLMs) have recently been shown to make effective use of limited training data, and prior work on DIFFA indicates that replacing an AR backbone with a diffusion counterpart can substantially improve audio understanding under matched settings, albeit at a proof-of-concept scale without large-scale instruction tuning, preference alignment, or practical decoding schemes. We introduce DIFFA-2, a practical diffusion-based LALM for general audio understanding. DIFFA-2 upgrades the speech encoder, employs dual semantic and acoustic adapters, and is trained with a four-stage curriculum that combines semantic and acoustic alignment, large-scale supervised fine-tuning, and variance-reduced preference optimization, using only fully open-source corpora. Experiments on MMSU, MMAU, and MMAR show that DIFFA-2 consistently improves over DIFFA and is competitive to strong AR LALMs under practical training budgets, supporting diffusion-based modeling is a viable backbone for large-scale audio understanding. Our code is available at https://github.com/NKU-HLT/DIFFA.git.