A Unified Deep Learning Framework for Short-Duration Speaker Verification in Adverse Environments

TL;DR

This paper presents a unified deep learning framework that enhances speaker verification robustness in adverse environments by integrating feature pyramid modules, self-adaptive VAD, and speech enhancement, effectively handling short speech, noise, reverberation, and long non-speech segments.

Contribution

It introduces a novel end-to-end deep learning framework combining FPM-based MSA, SAS-VAD, and speech enhancement, the first to unify these models for robust speaker verification.

Findings

Outperforms baseline i-vector and deep speaker embedding systems.

Effective in noisy and reverberant conditions on Korean indoor and VoxCeleb datasets.

Demonstrates robustness to short speech segments and long non-speech segments.

Abstract

Speaker verification (SV) has recently attracted considerable research interest due to the growing popularity of virtual assistants. At the same time, there is an increasing requirement for an SV system: it should be robust to short speech segments, especially in noisy and reverberant environments. In this paper, we consider one more important requirement for practical applications: the system should be robust to an audio stream containing long non-speech segments, where a voice activity detection (VAD) is not applied. To meet these two requirements, we introduce feature pyramid module (FPM)-based multi-scale aggregation (MSA) and self-adaptive soft VAD (SAS-VAD). We present the FPM-based MSA to deal with short speech segments in noisy and reverberant environments. Also, we use the SAS-VAD to increase the robustness to long non-speech segments. To further improve the robustness to acoustic distortions (i.e., noise and reverberation), we apply a masking-based speech enhancement (SE) method. We combine SV, VAD, and SE models in a unified deep learning framework and jointly train the entire network in an end-to-end manner. To the best of our knowledge, this is the first work combining these three models in a deep learning framework. We conduct experiments on Korean indoor (KID) and VoxCeleb datasets, which are corrupted by noise and reverberation. The results show that the proposed method is effective for SV in the challenging conditions and performs better than the baseline i-vector and deep speaker embedding systems.