Test-Time Training for Speech Enhancement

TL;DR

This paper applies Test-Time Training to speech enhancement, enabling models to adapt dynamically to new noise conditions and domain shifts during inference without labeled data, leading to improved speech quality.

Contribution

It introduces a novel TTT framework with self-supervised tasks for speech enhancement, demonstrating effective domain adaptation during inference.

Findings

Consistent improvements in speech quality metrics

Outperforms baseline models on synthetic and real-world datasets

Effective adaptation strategies balancing performance and efficiency

Abstract

This paper introduces a novel application of Test-Time Training (TTT) for Speech Enhancement, addressing the challenges posed by unpredictable noise conditions and domain shifts. This method combines a main speech enhancement task with a self-supervised auxiliary task in a Y-shaped architecture. The model dynamically adapts to new domains during inference time by optimizing the proposed self-supervised tasks like noise-augmented signal reconstruction or masked spectrogram prediction, bypassing the need for labeled data. We further introduce various TTT strategies offering a trade-off between adaptation and efficiency. Evaluations across synthetic and real-world datasets show consistent improvements across speech quality metrics, outperforming the baseline model. This work highlights the effectiveness of TTT in speech enhancement, providing insights for future research in adaptive and robust speech processing.