Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models

TL;DR

This paper investigates the Discrete Representation Inconsistency (DRI) in neural audio codecs, analyzes its impact on language models, and proposes mitigation techniques validated on large-scale speech datasets, improving audio generation consistency.

Contribution

It introduces a quantitative analysis of DRI in neural audio codecs and presents a method to effectively mitigate this inconsistency, enhancing speech generation quality.

Findings

DRI causes divergence in audio token sequences for the same audio.

Mitigation techniques significantly reduce inconsistencies in neural codec language models.

Experiments on LibriTTS and MLS datasets show improved audio generation quality.

Abstract

Building upon advancements in Large Language Models (LLMs), the field of audio processing has seen increased interest in training audio generation tasks with discrete audio token sequences. However, directly discretizing audio by neural audio codecs often results in sequences that fundamentally differ from text sequences. Unlike text, where text token sequences are deterministic, discrete audio tokens can exhibit significant variability based on contextual factors, while still producing perceptually identical audio segments. We refer to this phenomenon as \textbf{Discrete Representation Inconsistency (DRI)}. This inconsistency can lead to a single audio segment being represented by multiple divergent sequences, which creates confusion in neural codec language models and results in omissions and repetitions during speech generation. In this paper, we quantitatively analyze the DRI phenomenon within popular audio tokenizers such as EnCodec. Our approach effectively mitigates the DRI phenomenon of the neural audio codec. Furthermore, extensive experiments on the neural codec language model over LibriTTS and large-scale MLS datases (44,000 hours) demonstrate the effectiveness and generality of our method. The demo of audio samples is available online~\footnote{\url{https://consistencyinneuralcodec.github.io}}.