HASRD: Hierarchical Acoustic and Semantic Representation Disentanglement

TL;DR

HASRD introduces a hierarchical framework that disentangles semantic and acoustic representations in speech, improving ASR accuracy and reconstruction quality while reducing bitrate.

Contribution

It presents a novel hierarchical disentanglement method that enhances speech representation learning by separating semantic and acoustic tokens effectively.

Findings

44% relative WER improvement over SpeechTokenizer

Achieves high-quality reconstruction with lower bitrate

Enhances encoder efficiency without losing performance

Abstract

Effective speech representations for spoken language models must balance semantic relevance with acoustic fidelity for high-quality reconstruction. However, existing approaches struggle to achieve both simultaneously. To address this, we introduce Hierarchical Acoustic and Semantic Representation Disentanglement (HASRD, pronounced `hazard'), a framework that factorizes self-supervised learning representations into discrete semantic and acoustic tokens. HASRD assigns the semantic representation to the first codebook, while encoding acoustic residuals in subsequent codebooks. This preserves ASR performance while achieving high-quality reconstruction. Additionally, we enhance HASRD's encoder efficiency, improving ASR performance without compromising reconstruction quality. Compared to SpeechTokenizer, HASRD achieves a 44% relative WER improvement, superior reconstruction quality, and 2x lower bitrate, demonstrating its effectiveness in disentangling acoustic and semantic information.