Causally Disentangled Contrastive Learning for Multilingual Speaker Embeddings

TL;DR

This paper explores how demographic attributes like gender, age, and accent are encoded in self-supervised speaker embeddings and evaluates methods to reduce this leakage without significantly harming speaker verification performance.

Contribution

It introduces and compares two debiasing strategies—adversarial training and a causal bottleneck—for mitigating demographic information in speaker embeddings.

Findings

Gender information is strongly linearly encoded in embeddings.

Adversarial debiasing reduces gender leakage but affects verification accuracy.

Causal bottleneck suppresses demographic info but causes performance degradation.

Abstract

Self-supervised speaker embeddings are widely used in speaker verification systems, but prior work has shown that they often encode sensitive demographic attributes, raising fairness and privacy concerns. This paper investigates the extent to which demographic information, specifically gender, age, and accent, is present in SimCLR-trained speaker embeddings and whether such leakage can be mitigated without severely degrading speaker verification performance. We study two debiasing strategies: adversarial training through gradient reversal and a causal bottleneck architecture that explicitly separates demographic and residual information. Demographic leakage is quantified using both linear and nonlinear probing classifiers, while speaker verification performance is evaluated using ROC-AUC and EER. Our results show that gender information is strongly and linearly encoded in baseline embeddings, whereas age and accent are weaker and primarily nonlinearly represented. Adversarial debiasing reduces gender leakage but has limited effect on age and accent and introduces a clear trade-off with verification accuracy. The causal bottleneck further suppresses demographic information, particularly in the residual representation, but incurs substantial performance degradation. These findings highlight fundamental limitations in mitigating demographic leakage in self-supervised speaker embeddings and clarify the trade-offs inherent in current debiasing approaches.