Probing the Robustness Properties of Neural Speech Codecs

TL;DR

This paper systematically evaluates neural speech codecs' robustness to noise, analyzing their linearity and frequency response, and provides insights for improving their real-world performance.

Contribution

It offers a comprehensive analysis of neural speech codecs' robustness, linearity, and frequency response, highlighting factors affecting their generalization in noisy environments.

Findings

Neural speech codecs show varying robustness to noise.

Non-linear distortions partly explain robustness differences.

Frequency response analysis reveals factors impacting audio fidelity.

Abstract

Neural speech codecs have revolutionized speech coding, achieving higher compression while preserving audio fidelity. Beyond compression, they have emerged as tokenization strategies, enabling language modeling on speech and driving paradigm shifts across various speech processing tasks. Despite these advancements, their robustness in noisy environments remains underexplored, raising concerns about their generalization to real-world scenarios. In this work, we systematically evaluate neural speech codecs under various noise conditions, revealing non-trivial differences in their robustness. We further examine their linearity properties, uncovering non-linear distortions which partly explain observed variations in robustness. Lastly, we analyze their frequency response to identify factors affecting audio fidelity. Our findings provide critical insights into codec behavior and future codec design, as well as emphasizing the importance of noise robustness for their real-world integration.