Evaluating Identity Leakage in Speaker De-Identification Systems

TL;DR

This paper introduces a benchmark to measure residual speaker identity leakage in de-identification systems, revealing that current methods still pose significant privacy risks despite efforts to anonymize speech.

Contribution

It proposes a comprehensive benchmark with multiple metrics to evaluate identity leakage, exposing the limitations of existing speaker de-identification systems.

Findings

All evaluated systems leak identity information.

The best system performs only slightly better than random guessing.

The worst system has a 45% hit rate within top 50 candidates.

Abstract

Speaker de-identification aims to conceal a speaker's identity while preserving intelligibility of the underlying speech. We introduce a benchmark that quantifies residual identity leakage with three complementary error rates: equal error rate, cumulative match characteristic hit rate, and embedding-space similarity measured via canonical correlation analysis and Procrustes analysis. Evaluation results reveal that all state-of-the-art speaker de-identification systems leak identity information. The highest performing system in our evaluation performs only slightly better than random guessing, while the lowest performing system achieves a 45% hit rate within the top 50 candidates based on CMC. These findings highlight persistent privacy risks in current speaker de-identification technologies.