Universal Robust Speech Adaptation for Cross-Domain Speech Recognition and Enhancement

TL;DR

URSA-GAN is a unified generative framework that enhances cross-domain speech recognition and enhancement by using dual-embeddings and stochastic perturbation to improve robustness against unseen noise and channel distortions.

Contribution

This paper introduces URSA-GAN, a novel domain-aware GAN-based model with dual-embeddings and stochastic perturbation for robust speech adaptation across diverse conditions.

Findings

Reduces character error rates in ASR by 16.16%.

Improves perceptual metrics in speech enhancement by 15.58%.

Demonstrates strong generalization to unseen noise and channel conditions.

Abstract

Pre-trained models for automatic speech recognition (ASR) and speech enhancement (SE) have exhibited remarkable capabilities under matched noise and channel conditions. However, these models often suffer from severe performance degradation when confronted with domain shifts, particularly in the presence of unseen noise and channel distortions. In view of this, we in this paper present URSA-GAN, a unified and domain-aware generative framework specifically designed to mitigate mismatches in both noise and channel conditions. URSA-GAN leverages a dual-embedding architecture that consists of a noise encoder and a channel encoder, each pre-trained with limited in-domain data to capture domain-relevant representations. These embeddings condition a GAN-based speech generator, facilitating the synthesis of speech that is acoustically aligned with the target domain while preserving phonetic content. To enhance generalization further, we propose dynamic stochastic perturbation, a novel regularization technique that introduces controlled variability into the embeddings during generation, promoting robustness to unseen domains. Empirical results demonstrate that URSA-GAN effectively reduces character error rates in ASR and improves perceptual metrics in SE across diverse noisy and mismatched channel scenarios. Notably, evaluations on compound test conditions with both channel and noise degradations confirm the generalization ability of URSA-GAN, yielding relative improvements of 16.16% in ASR performance and 15.58% in SE metrics.