Robust One-step Speech Enhancement via Consistency Distillation

TL;DR

This paper introduces ROSE-CD, a novel one-step speech enhancement model using consistency distillation with randomized trajectories, achieving faster inference and superior performance over multi-step diffusion models.

Contribution

It presents the first pure one-step consistency distillation approach for diffusion-based speech enhancement, improving robustness and performance.

Findings

Achieves 54x faster inference speed than the 30-step teacher model.

Surpasses teacher model performance in speech quality metrics.

Demonstrates strong generalization on out-of-domain and real-world noisy data.

Abstract

Diffusion models have shown strong performance in speech enhancement, but their real-time applicability has been limited by multi-step iterative sampling. Consistency distillation has recently emerged as a promising alternative by distilling a one-step consistency model from a multi-step diffusion-based teacher model. However, distilled consistency models are inherently biased towards the sampling trajectory of the teacher model, making them less robust to noise and prone to inheriting inaccuracies from the teacher model. To address this limitation, we propose ROSE-CD: Robust One-step Speech Enhancement via Consistency Distillation, a novel approach for distilling a one-step consistency model. Specifically, we introduce a randomized learning trajectory to improve the model's robustness to noise. Furthermore, we jointly optimize the one-step model with two time-domain auxiliary losses, enabling it to recover from teacher-induced errors and surpass the teacher model in overall performance. This is the first pure one-step consistency distillation model for diffusion-based speech enhancement, achieving 54 times faster inference speed and superior performance compared to its 30-step teacher model. Experiments on the VoiceBank-DEMAND dataset demonstrate that the proposed model achieves state-of-the-art performance in terms of speech quality. Moreover, its generalization ability is validated on both an out-of-domain dataset and real-world noisy recordings.