Enhancing Speaker Verification with Whispered Speech via Post-Processing

TL;DR

This paper introduces a post-processing model that significantly improves speaker verification accuracy with whispered speech, addressing challenges posed by acoustic differences and noise.

Contribution

The proposed encoder-decoder system, built on a fine-tuned backbone, enhances whispered speech verification, achieving notable relative improvements over existing methods.

Findings

22.26% relative improvement in whispered vs normal speech trials

Achieved 98.16% AUC in normal vs whispered speech verification

Attained 1.88% EER in whispered vs whispered speech, outperforming prior models

Abstract

Speaker verification is a task of confirming an individual's identity through the analysis of their voice. Whispered speech differs from phonated speech in acoustic characteristics, which degrades the performance of speaker verification systems in real-life scenarios, including avoiding fully phonated speech to protect privacy, disrupt others, or when the lack of full vocalization is dictated by a disease. In this paper we propose a model with a training recipe to obtain more robust representations against whispered speech hindrances. The proposed system employs an encoder--decoder structure built atop a fine-tuned speaker verification backbone, optimized jointly using cosine similarity--based classification and triplet loss. We gain relative improvement of 22.26\% compared to the baseline (baseline 6.77\% vs ours 5.27\%) in normal vs whispered speech trials, achieving AUC of 98.16\%. In tests comparing whispered to whispered, our model attains an EER of 1.88\% with AUC equal to 99.73\%, which represents a 15\% relative enhancement over the prior leading ReDimNet-B2. We also offer a summary of the most popular and state-of-the-art speaker verification models in terms of their performance with whispered speech. Additionally, we evaluate how these models perform under noisy audios, obtaining that generally the same relative level of noise degrades the performance of speaker verification more significantly on whispered speech than on normal speech.