DAME: Duration-Aware Matryoshka Embedding for Duration-Robust Speaker Verification

TL;DR

DAME introduces a duration-aware hierarchical embedding framework for speaker verification, improving accuracy on short utterances without extra inference cost by capturing duration-specific speaker traits.

Contribution

The paper proposes a novel nested hierarchy embedding approach that adapts to utterance duration, enhancing short-utterance speaker verification performance.

Findings

Reduces EER on short-duration trials across multiple datasets.

Maintains full-length performance without additional inference cost.

Works with various encoder architectures and training setups.

Abstract

Short-utterance speaker verification remains challenging due to limited speaker-discriminative cues in short speech segments. While existing methods focus on enhancing speaker encoders, the embedding learning strategy still forces a single fixed-dimensional representation reused for utterances of any length, leaving capacity misaligned with the information available at different durations. We propose Duration-Aware Matryoshka Embedding (DAME), a model-agnostic framework that builds a nested hierarchy of sub-embeddings aligned to utterance durations: lower-dimensional representations capture compact speaker traits from short utterances, while higher dimensions encode richer details from longer speech. DAME supports both training from scratch and fine-tuning, and serves as a direct alternative to conventional large-margin fine-tuning, consistently improving performance across durations. On the VoxCeleb1-O/E/H and VOiCES evaluation sets, DAME consistently reduces the equal error rate on 1-s and other short-duration trials, while maintaining full-length performance with no additional inference cost. These gains generalize across various speaker encoder architectures under both general training and fine-tuning setups.