Khala: Scaling Acoustic Token Language Models Toward High-Fidelity Music Generation

TL;DR

This paper introduces a unified acoustic-token hierarchy for music generation, enabling high-fidelity output through a two-stage coarse-to-fine modeling process within a single deep representation space.

Contribution

It proposes a novel 64-layer residual vector quantization framework with hybrid-attention training, demonstrating that structure and detail can be modeled jointly without separate semantic stages.

Findings

Text-vocal alignment can emerge without a separate semantic token stage.

Initializing super-resolution from the backbone improves convergence and quality.

A fixed 62-step inference process efficiently refines music tokens.

Abstract

A common design pattern in high-quality music generation is to handle structure and fidelity in different representation spaces: a generator first models high-level structure, followed by diffusion-based or neural decoding stages that reconstruct fine details. In this work, we explore an alternative view: both may be progressively modeled within a single deep acoustic-token hierarchy. To study this, we build a 64-layer residual vector quantization (RVQ) acoustic representation and propose a two-stage coarse-to-fine generation framework. A backbone model first generates coarse acoustic tokens for the full track, and a super-resolution model then completes finer tokens within the same acoustic token space. The super-resolution stage works at full-track scale and refines tokens layer by layer while running in parallel over time, leading to a fixed 62-step inference process. To jointly improve lyric alignment and fine-detail reconstruction, we further introduce hybrid-attention training: the alignment objective uses causal attention, while layer-wise refinement uses full attention. A key finding is that text--vocal alignment can emerge within pure acoustic-token language modeling, without requiring a separate semantic token stage. Moreover, initializing the super-resolution model from the trained backbone significantly improves convergence and final quality. Taken together, our results suggest that high-quality music generation can be effectively pursued without separating structure and fidelity into heterogeneous representation spaces. Instead, both can be progressively modeled within a unified acoustic-token hierarchy, pointing toward a simpler and more unified path to high-quality music generation.