DisSR: Disentangling Speech Representation for Degradation-Prior Guided Cross-Domain Speech Restoration

TL;DR

DisSR introduces a general speech restoration model that leverages degradation priors and domain adaptation to effectively restore speech across various distortions and unseen domains, outperforming single-task models.

Contribution

The paper proposes DisSR, a novel disentangling-based speech restoration framework that incorporates degradation-prior guidance and cross-domain training for improved generalization.

Findings

Achieves high-quality speech restoration across multiple distortion types.

Demonstrates superior generalization to unseen domains.

Outperforms existing single-task speech restoration models.

Abstract

Previous speech restoration (SR) primarily focuses on single-task speech restoration (SSR), which cannot address general speech restoration problems. Training specific SSR models for different distortions is time-consuming and lacks generality. In addition, most studies ignore the problem of model generalization across unseen domains. To overcome those limitations, we propose DisSR, a Disentangling Speech Representation based general speech restoration model with two properties: 1) Degradation-prior guidance, which extracts speaker-invariant degradation representation to guide the diffusion-based speech restoration model. 2) Domain adaptation, where we design cross-domain alignment training to enhance the model's adaptability and generalization on cross-domain data, respectively. Experimental results demonstrate that our method can produce high-quality restored speech under various distortion conditions. Audio samples can be found at https://itspsp.github.io/DisSR.