NASTAR: Noise Adaptive Speech Enhancement with Target-Conditional Resampling

TL;DR

NASTAR introduces a one-shot noise adaptation method for speech enhancement that uses noise extraction and retrieval to improve performance in target environments with minimal data.

Contribution

This work is the first to combine noise extraction and retrieval for one-shot noise adaptation in speech enhancement models.

Findings

NASTAR effectively adapts to target noise with only one noisy sample.

Both noise extractor and retrieval model improve adaptation performance.

Experimental results demonstrate significant enhancement improvements.

Abstract

For deep learning-based speech enhancement (SE) systems, the training-test acoustic mismatch can cause notable performance degradation. To address the mismatch issue, numerous noise adaptation strategies have been derived. In this paper, we propose a novel method, called noise adaptive speech enhancement with target-conditional resampling (NASTAR), which reduces mismatches with only one sample (one-shot) of noisy speech in the target environment. NASTAR uses a feedback mechanism to simulate adaptive training data via a noise extractor and a retrieval model. The noise extractor estimates the target noise from the noisy speech, called pseudo-noise. The noise retrieval model retrieves relevant noise samples from a pool of noise signals according to the noisy speech, called relevant-cohort. The pseudo-noise and the relevant-cohort set are jointly sampled and mixed with the source speech corpus to prepare simulated training data for noise adaptation. Experimental results show that NASTAR can effectively use one noisy speech sample to adapt an SE model to a target condition. Moreover, both the noise extractor and the noise retrieval model contribute to model adaptation. To our best knowledge, NASTAR is the first work to perform one-shot noise adaptation through noise extraction and retrieval.