U3-xi: Pushing the Boundaries of Speaker Recognition by Incorporating Uncertainty

TL;DR

This paper introduces U3-xi, a novel framework that enhances speaker recognition by estimating and utilizing frame-level uncertainty to improve the reliability and interpretability of speaker embeddings, leading to significant performance gains.

Contribution

U3-xi is the first framework to incorporate comprehensive uncertainty estimation into speaker recognition, combining multiple supervision strategies and a Transformer-based module for improved accuracy.

Findings

Achieves 21.1% relative EER reduction on VoxCeleb1

Attains 15.57% relative minDCF improvement

Demonstrates model-agnostic applicability and robustness

Abstract

An utterance-level speaker embedding is typically obtained by aggregating a sequence of frame-level representations. However, in real-world scenarios, individual frames encode not only speaker-relevant information but also various nuisance factors. As a result, different frames contribute unequally to the final utterance-level speaker representation for Automatic Speaker Verification systems. To address this issue, we propose to estimate the inherent uncertainty of each frame and assign adaptive weights accordingly, where frames with higher uncertainty receive lower attention. Based on this idea, we present U3-xi, a comprehensive framework designed to produce more reliable and interpretable uncertainty estimates for speaker embeddings. Specifically, we introduce several strategies for uncertainty supervision. First, we propose speaker-level uncertainty supervision via a Stochastic Variance Loss, where the distance between an utterance embedding and its corresponding speaker centroid serves as a pseudo ground truth for uncertainty learning. Second, we incorporate global-level uncertainty supervision by injecting the predicted uncertainty into the sof tmax scale during training. This adaptive scaling mechanism adjusts the sharpness of the decision boundary according to sample difficulty, providing global guidance. Third, we redesign the uncertainty estimation module by integrating a Transformer encoder with multi-view self-attention, enabling the model to capture rich local and long-range temporal dependencies. Comprehensive experiments demonstrate that U3-xi is model-agnostic and can be seamlessly applied to various speaker encoders. In particular, when applied to ECAPA-TDNN, it achieves 21.1% and 15.57% relative improvements on the VoxCeleb1 test sets in terms of EER and minDCF, respectively.