Privacy-oriented manipulation of speaker representations

TL;DR

This paper introduces a novel method using a Vector-Quantized Variational Autoencoder with adversarial training and mutual information loss to remove private attributes like age and sex from speaker embeddings, enhancing privacy.

Contribution

It presents a new approach for manipulating speaker embeddings to protect private information, combining VQ-VAE, adversarial classifiers, and mutual information loss.

Findings

Effective removal of sex and age information from embeddings.

Robust performance against both ignorant and informed attackers.

Applicable to in-domain and out-of-domain data.

Abstract

Speaker embeddings are ubiquitous, with applications ranging from speaker recognition and diarization to speech synthesis and voice anonymisation. The amount of information held by these embeddings lends them versatility, but also raises privacy concerns. Speaker embeddings have been shown to contain information on age, sex, health and more, which speakers may want to keep private, especially when this information is not required for the target task. In this work, we propose a method for removing and manipulating private attributes from speaker embeddings that leverages a Vector-Quantized Variational Autoencoder architecture, combined with an adversarial classifier and a novel mutual information loss. We validate our model on two attributes, sex and age, and perform experiments with ignorant and fully-informed attackers, and with in-domain and out-of-domain data.