STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecs

TL;DR

STACodec is a novel audio codec that effectively balances acoustic fidelity and semantic information by integrating semantic tokens through a unified framework, improving both reconstruction quality and semantic task performance.

Contribution

The paper introduces STACodec, a unified codec that incorporates semantic information via semantic token assignment, and proposes a semantic pre-distillation module to enhance efficiency and performance.

Findings

Outperforms existing hybrid codecs in audio reconstruction.

Achieves better balance between fidelity and semantic capability.

Improves downstream semantic task performance.

Abstract

Neural audio codecs are widely used for audio compression and can be integrated into token-based language models. Traditional codecs preserve acoustic details well but lack semantic information. Recent hybrid codecs attempt to incorporate semantic information through distillation, but this often degrades reconstruction performance, making it difficult to achieve both. To address this limitation, we introduce STACodec, a unified codec that integrates semantic information from self-supervised learning (SSL) models into the first layer of residual vector quantization (RVQ-1) via semantic token assignment (STA). To further eliminate reliance on SSL-based semantic tokenizers and improve efficiency during inference, we propose a semantic pre-distillation (SPD) module, which predicts semantic tokens directly for assignment to the first RVQ layer during inference. Experimental results show that STACodec outperforms existing hybrid codecs in both audio reconstruction and downstream semantic tasks, demonstrating a better balance between acoustic fidelity and semantic capability.