Towards General Discrete Speech Codec for Complex Acoustic Environments: A Study of Reconstruction and Downstream Task Consistency

TL;DR

This paper introduces ERSB, a benchmark to evaluate neural speech codecs' robustness in complex environments, revealing current limitations in reconstruction quality and downstream task consistency.

Contribution

The study presents a new benchmark for assessing environment resilience of neural speech codecs and systematically evaluates their performance in complex acoustic settings.

Findings

Complex environments degrade reconstruction quality.

Downstream task consistency is compromised in noisy settings.

Current codecs lack robustness in real-world scenarios.

Abstract

Neural speech codecs excel in reconstructing clean speech signals; however, their efficacy in complex acoustic environments and downstream signal processing tasks remains underexplored. In this study, we introduce a novel benchmark named Environment-Resilient Speech Codec Benchmark (ERSB) to systematically evaluate whether neural speech codecs are environment-resilient. Specifically, we assess two key capabilities: (1) robust reconstruction, which measures the preservation of both speech and non-speech acoustic details, and (2) downstream task consistency, which ensures minimal deviation in downstream signal processing tasks when using reconstructed speech instead of the original. Our comprehensive experiments reveal that complex acoustic environments significantly degrade signal reconstruction and downstream task consistency. This work highlights the limitations of current speech codecs and raises a future direction that improves them for greater environmental resilience.