ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers

TL;DR

ESC introduces a lightweight, transformer-based speech codec that uses cross-scale residual vector quantization for efficient, high-quality speech reconstruction with lower complexity and improved bitrate efficiency.

Contribution

The paper presents a novel transformer-based speech codec employing cross-scale residual vector quantization and hierarchical decoding, reducing model complexity while maintaining high audio quality.

Findings

Achieves high-fidelity speech reconstruction with fewer parameters.

Outperforms existing codecs in bitrate efficiency.

Demonstrates the effectiveness of hierarchical transformers in speech coding.

Abstract

Neural speech codecs aim to compress input signals into minimal bits while maintaining content quality in a low-latency manner. However, existing neural codecs often trade model complexity for reconstruction performance. These codecs primarily use convolutional blocks for feature transformation, which are not inherently suited for capturing the local redundancies in speech signals. To compensate, they require either adversarial discriminators or a large number of model parameters to enhance audio quality. In response to these challenges, we introduce the Efficient Speech Codec (ESC), a lightweight, parameter-efficient speech codec based on a cross-scale residual vector quantization scheme and transformers. Our model employs mirrored hierarchical window transformer blocks and performs step-wise decoding from coarse-to-fine feature representations. To enhance bitrate efficiency, we propose a novel combination of vector quantization techniques along with a pre-training paradigm. Extensive experiments demonstrate that ESC can achieve high-fidelity speech reconstruction with significantly lower model complexity, making it a promising alternative to existing convolutional audio codecs.