UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstruction

TL;DR

UniSRCodec is a novel single-codebook neural audio codec that achieves high-fidelity, low-bitrate audio compression across a wide frequency range by combining time-frequency compression, sub-band reconstruction, and phase recovery, outperforming existing methods.

Contribution

It introduces UniSRCodec, a unified single-codebook audio codec supporting high sampling rates and fidelity, with innovative sub-band reconstruction and phase recovery techniques.

Findings

Achieves state-of-the-art performance among cross-domain single-codebook codecs.

Maintains high reconstruction quality at only 40 tokens bitrate.

Comparable to multi-codebook methods in quality.

Abstract

Neural Audio Codecs (NACs) can reduce transmission overhead by performing compact compression and reconstruction, which also aim to bridge the gap between continuous and discrete signals. Existing NACs can be divided into two categories: multi-codebook and single-codebook codecs. Multi-codebook codecs face challenges such as structural complexity and difficulty in adapting to downstream tasks, while single-codebook codecs, though structurally simpler, suffer from low-fidelity, ineffective modeling of unified audio, and an inability to support modeling of high-frequency audio. We propose the UniSRCodec, a single-codebook codec capable of supporting high sampling rate, low-bandwidth, high fidelity, and unified. We analyze the inefficiency of waveform-based compression and introduce the time and frequency compression method using the Mel-spectrogram, and cooperate with a Vocoder to recover the phase information of the original audio. Moreover, we propose a sub-band reconstruction technique to achieve high-quality compression across both low and high frequency bands. Subjective and objective experimental results demonstrate that UniSRCodec achieves state-of-the-art (SOTA) performance among cross-domain single-codebook codecs with only a token rate of 40, and its reconstruction quality is comparable to that of certain multi-codebook methods. Our demo page is available at https://wxzyd123.github.io/unisrcodec.