UniAudio 2.0: A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization

TL;DR

UniAudio 2.0 introduces a unified audio language model with a novel reasoning-based audio tokenizer and an autoregressive architecture, enabling effective understanding, generation, and zero-shot generalization across diverse audio tasks.

Contribution

It proposes ReasoningCodec, a discrete audio tokenizer with high-level reasoning and high-fidelity reconstruction, and a unified autoregressive model trained on large-scale text and audio data.

Findings

Achieves understanding comparable to continuous representations.

Improves audio generation quality and reconstruction fidelity.

Demonstrates strong few-shot and zero-shot generalization.

Abstract

We study two foundational problems in audio language models: (1) how to design an audio tokenizer that can serve as an intermediate representation for both understanding and generation; and (2) how to build an audio foundation model that generalizes in few-shot and zero-shot settings, analogous to large language models. To this end, we make the following two contributions. First, we propose ReasoningCodec, a discrete audio codec that factorizes audio into (i) reasoning tokens, which encode text-aligned, high-level analysis and planning representations for audio understanding and hierarchical generation, and (ii) reconstruction tokens, which encode semantic-rich acoustic cues for high-fidelity waveform reconstruction. This design achieves understanding performance comparable to strong continuous representations while improving generation quality and reconstruction fidelity over prior discrete tokenizers. Second, we introduce a unified autoregressive architecture for text and audio, together with multi-stage training and multi-task data construction. Using this framework, we train UniAudio 2.0 on 100B text tokens and 60B audio tokens. Across a wide range of speech, sound, and music tasks, UniAudio 2.0 performs competitively on in-domain evaluations and demonstrates strong few-shot and zero-shot generalization to unseen tasks. Demo, code, and checkpoints will be available at \href{https://dongchaoyang.top/UniAudio2Demo/}{https://dongchaoyang.top/UniAudio2Demo/}.