Improving generalizability of distilled self-supervised speech processing models under distorted settings

TL;DR

This paper enhances the robustness of distilled self-supervised speech models in distorted environments by applying domain adaptation techniques during knowledge distillation, leading to improved performance across various tasks without increasing model size.

Contribution

It introduces Cross-Distortion Mapping and Domain Adversarial Training into the distillation process to address domain mismatch issues in distorted speech environments.

Findings

Consistent performance improvements in distorted settings

Effective domain adaptation during knowledge distillation

Maintains efficient model size while improving robustness

Abstract

Self-supervised learned (SSL) speech pre-trained models perform well across various speech processing tasks. Distilled versions of SSL models have been developed to match the needs of on-device speech applications. Though having similar performance as original SSL models, distilled counterparts suffer from performance degradation even more than their original versions in distorted environments. This paper proposes to apply Cross-Distortion Mapping and Domain Adversarial Training to SSL models during knowledge distillation to alleviate the performance gap caused by the domain mismatch problem. Results show consistent performance improvements under both in- and out-of-domain distorted setups for different downstream tasks while keeping efficient model size.