A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generation

TL;DR

This paper investigates methods to improve waveform resynthesis from neural audio codec tokens, highlighting the impact of learning targets and introducing a Schr"odinger Bridge approach for enhanced speech quality.

Contribution

It introduces a novel Schr"odinger Bridge-based resynthesis method and analyzes how different strategies influence audio quality and perception.

Findings

Schr"odinger Bridge improves waveform reconstruction quality.

Choice of learning target significantly affects audio perception.

Different resynthesis strategies impact both machine and human evaluations.

Abstract

Neural Audio Codecs, initially designed as a compression technique, have gained more attention recently for speech generation. Codec models represent each audio frame as a sequence of tokens, i.e., discrete embeddings. The discrete and low-frequency nature of neural codecs introduced a new way to generate speech with token-based models. As these tokens encode information at various levels of granularity, from coarse to fine, most existing works focus on how to better generate the coarse tokens. In this paper, we focus on an equally important but often overlooked question: How can we better resynthesize the waveform from coarse tokens? We point out that both the choice of learning target and resynthesis approach have a dramatic impact on the generated audio quality. Specifically, we study two different strategies based on token prediction and regression, and introduce a new method based on Schr\"odinger Bridge. We examine how different design choices affect machine and human perception.