Autoregressive Speech Enhancement via Acoustic Tokens

TL;DR

This paper explores using acoustic tokens with autoregressive models for speech enhancement, demonstrating improved speaker identity preservation and performance over semantic tokens, but still lagging behind continuous representations.

Contribution

It introduces a transducer-based autoregressive architecture for acoustic token-based speech enhancement and provides a comprehensive performance analysis of discrete representations.

Findings

Acoustic tokens outperform semantic tokens in preserving speaker identity.

Autoregressive models further improve speech enhancement performance.

Discrete representations still underperform compared to continuous ones.

Abstract

In speech processing pipelines, improving the quality and intelligibility of real-world recordings is crucial. While supervised regression is the primary method for speech enhancement, audio tokenization is emerging as a promising alternative for a smooth integration with other modalities. However, research on speech enhancement using discrete representations is still limited. Previous work has mainly focused on semantic tokens, which tend to discard key acoustic details such as speaker identity. Additionally, these studies typically employ non-autoregressive models, assuming conditional independence of outputs and overlooking the potential improvements offered by autoregressive modeling. To address these gaps we: 1) conduct a comprehensive study of the performance of acoustic tokens for speech enhancement, including the effect of bitrate and noise strength; 2) introduce a novel transducer-based autoregressive architecture specifically designed for this task. Experiments on VoiceBank and Libri1Mix datasets show that acoustic tokens outperform semantic tokens in terms of preserving speaker identity, and that our autoregressive approach can further improve performance. Nevertheless, we observe that discrete representations still fall short compared to continuous ones, highlighting the need for further research in this area.