Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction

TL;DR

This paper introduces a novel continuous-valued token modeling approach for audio generation using a causal language model with token-wise diffusion, achieving significant improvements over previous discrete methods and state-of-the-art diffusion models.

Contribution

It presents a new continuous token modeling method with diffusion and a masked next-token prediction task, enhancing audio generation quality and efficiency.

Findings

20% and 40% improvements in FAD and KL divergence over previous methods.

41% and 33% FAD improvements with fewer parameters.

Comparable performance to state-of-the-art diffusion models.

Abstract

Autoregressive next-token prediction with the Transformer decoder has become a de facto standard in large language models (LLMs), achieving remarkable success in Natural Language Processing (NLP) at scale. Extending this paradigm to audio poses unique challenges due to its inherently continuous nature. We research audio generation with a causal language model (LM) without discrete tokens. We leverage token-wise diffusion to model the continuous distribution of the next continuous-valued token. Our approach delivers significant improvements over previous discrete solution, AudioGen, achieving 20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL) divergence, respectively. Additionally, we propose a novel masked next-token prediction task that incorporates masked prediction into the causal LM framework. On AudioCaps, the innovation yields 41% and 33% relative FAD improvements over AudioGen Base (285M) and AudioGen Large (1B) models, respectively, and is on par with the state-of-the-art (SOTA) diffusion models. Furthermore, we achieve these results with significantly fewer parameters -- 193M for our Base and 462M for our Large models.