Lightweight speech enhancement guided target speech extraction in noisy multi-speaker scenarios

TL;DR

This paper introduces a lightweight speech enhancement guided target speech extraction model, GTCRN, that improves performance in noisy multi-speaker scenarios through novel extensions and training strategies.

Contribution

The paper proposes LGTSE and D-LGTSE extensions to enhance TSE robustness in noisy environments, along with a two-stage training strategy for better performance.

Findings

Achieved 0.89 dB SISDR improvement on Libri2Mix

Improved PESQ by 0.16 and STOI by 1.97%

Validated effectiveness in noisy multi-speaker scenarios

Abstract

Target speech extraction (TSE) has achieved strong performance in relatively simple conditions such as one-speaker-plus-noise and two-speaker mixtures, but its performance remains unsatisfactory in noisy multi-speaker scenarios. To address this issue, we introduce a lightweight speech enhancement model, GTCRN, to better guide TSE in noisy environments. Building on our competitive previous speaker embedding/encoder-free framework SEF-PNet, we propose two extensions: LGTSE and D-LGTSE. LGTSE incorporates noise-agnostic enrollment guidance by denoising the input noisy speech before context interaction with enrollment speech, thereby reducing noise interference. D-LGTSE further improves system robustness against speech distortion by leveraging denoised speech as an additional noisy input during training, expanding the dynamic range of noisy conditions and enabling the model to directly learn from distorted signals. Furthermore, we propose a two-stage training strategy, first with GTCRN enhancement-guided pre-training and then joint fine-tuning, to fully exploit model potential.Experiments on the Libri2Mix dataset demonstrate significant improvements of 0.89 dB in SISDR, 0.16 in PESQ, and 1.97% in STOI, validating the effectiveness of our approach.