Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility

TL;DR

This paper introduces MaskSR2, an improved speech restoration model that combines semantic knowledge distillation with masked acoustic modeling, significantly enhancing speech intelligibility and quality without increasing model complexity.

Contribution

The paper proposes a novel method that integrates semantic representations into MaskSR, leading to substantial improvements in speech intelligibility and quality over existing models.

Findings

MaskSR2 reduces word error rate compared to MaskSR.

MaskSR2 achieves superior speech quality.

Semantic features significantly improve intelligibility.

Abstract

Speech restoration aims at restoring full-band speech with high quality and intelligibility, considering a diverse set of distortions. MaskSR is a recently proposed generative model for this task. As other models of its kind, MaskSR attains high quality but, as we show, intelligibility can be substantially improved. We do so by boosting the speech encoder component of MaskSR with predictions of semantic representations of the target speech, using a pre-trained self-supervised teacher model. Then, a masked language model is conditioned on the learned semantic features to predict acoustic tokens that encode low level spectral details of the target speech. We show that, with the same MaskSR model capacity and inference time, the proposed model, MaskSR2, significantly reduces the word error rate, a typical metric for intelligibility. MaskSR2 also achieves competitive word error rate among other models, while providing superior quality. An ablation study shows the effectiveness of various semantic representations.