Koopman Regularized Deep Speech Disentanglement for Speaker Verification

TL;DR

This paper introduces DKSD-AE, a novel autoencoder that uses Koopman operator learning and instance normalization to effectively disentangle speaker and content features in speech, improving speaker verification without relying on textual labels.

Contribution

The paper presents a structured autoencoder leveraging Koopman operator theory for disentangling speaker and content dynamics, reducing parameter count and supervision needs.

Findings

Achieves competitive speaker verification performance.

Maintains high content EER, confirming effective disentanglement.

Operates with fewer parameters and no textual supervision.

Abstract

Human speech contains both linguistic content and speaker dependent characteristics making speaker verification a key technology in identity critical applications. Modern deep learning speaker verification systems aim to learn speaker representations that are invariant to semantic content and nuisance factors such as ambient noise. However, many existing approaches depend on labelled data, textual supervision or large pretrained models as feature extractors, limiting scalability and practical deployment, raising sustainability concerns. We propose Deep Koopman Speech Disentanglement Autoencoder (DKSD-AE), a structured autoencoder that combines a novel multi-step Koopman operator learning module with instance normalization to disentangle speaker and content dynamics. Quantitative experiments across multiple datasets demonstrate that DKSD-AE achieves improved or competitive speaker verification performance compared to state-of-the-art baselines while maintaining high content EER, confirming effective disentanglement. These results are obtained with substantially fewer parameters and without textual supervision. Moreover, performance remains stable under increased evaluation scale, highlighting representation robustness and generalization. Our findings suggest that Koopman-based temporal modelling, when combined with instance normalization, provides an efficient and principled solution for speaker-focused representation learning.