EASY: Emotion-aware Speaker Anonymization via Factorized Distillation

TL;DR

EASY is a novel speaker anonymization framework that disentangles speaker identity, linguistic content, and emotional state to enhance privacy while preserving speech emotion and content.

Contribution

It introduces a sequential disentanglement process with factorized distillation to preserve emotion and content, advancing privacy protection in speech anonymization.

Findings

Outperforms baseline systems in privacy protection

Preserves emotional state and linguistic content effectively

Demonstrates robustness on VoicePrivacy Challenge datasets

Abstract

Emotion plays a significant role in speech interaction, conveyed through tone, pitch, and rhythm, enabling the expression of feelings and intentions beyond words to create a more personalized experience. However, most existing speaker anonymization systems employ parallel disentanglement methods, which only separate speech into linguistic content and speaker identity, often neglecting the preservation of the original emotional state. In this study, we introduce EASY, an emotion-aware speaker anonymization framework. EASY employs a novel sequential disentanglement process to disentangle speaker identity, linguistic content, and emotional representation, modeling each speech attribute in distinct subspaces through a factorized distillation approach. By independently constraining speaker identity and emotional representation, EASY minimizes information leakage, enhancing privacy protection while preserving original linguistic content and emotional state. Experimental results on the VoicePrivacy Challenge official datasets demonstrate that our proposed approach outperforms all baseline systems, effectively protecting speaker privacy while maintaining linguistic content and emotional state.