Schr\"odinger Bridge Consistency Trajectory Models for Speech Enhancement

TL;DR

This paper introduces Schr"odinger bridge Consistency Trajectory Models (SBCTM) for speech enhancement, significantly accelerating inference speed while maintaining high speech quality by applying consistency trajectory techniques and novel loss functions.

Contribution

The paper proposes SBCTM, combining Schr"odinger bridge and consistency trajectory models, with a new auxiliary loss, to improve inference speed and quality in speech enhancement.

Findings

Achieves approximately 16x faster inference than traditional Schr"odinger bridge models.

Maintains a favorable quality-speed trade-off with limited multi-step refinement.

Provides open-source code, models, and audio samples for reproducibility.

Abstract

Speech enhancement (SE) utilizing diffusion models is a promising technology that improves speech quality in noisy speech data. Furthermore, the Schr\"odinger bridge (SB) has recently been used in diffusion-based SE to improve speech quality by resolving a mismatch between the endpoint of the forward process and the starting point of the reverse process. However, the SB still exhibits slow inference owing to the necessity of a large number of function evaluations (NFE) for inference to obtain high-quality results. While Consistency Models (CMs) address this issue by employing consistency training that uses distillation from pretrained models in the field of image generation, it does not improve generation quality when the number of steps increases. As a solution to this problem, Consistency Trajectory Models (CTMs) not only accelerate inference speed but also maintain a favorable trade-off between quality and speed. Furthermore, SoundCTM demonstrates the applicability of CTM techniques to the field of sound generation. In this paper, we present Schr\"odinger bridge Consistency Trajectory Models (SBCTM) by applying the CTM's technique to the Schr\"odinger bridge for SE. Additionally, we introduce a novel auxiliary loss, including a perceptual loss, into the original CTM's training framework. As a result, SBCTM achieves an approximately 16x improvement in the real-time factor (RTF) compared to the conventional Schr\"odinger bridge for SE. Furthermore, the favorable trade-off between quality and speed in SBCTM allows for time-efficient inference by limiting multi-step refinement to cases where 1-step inference is insufficient. Our code, pretrained models, and audio samples are available at https://github.com/sony/sbctm/.