Evaluating voice anonymisation using similarity rank disclosure

TL;DR

This paper explores the use of similarity rank disclosure (SRD), an information-theoretic metric, for evaluating voice anonymisation systems, offering a threshold-independent and more comprehensive privacy assessment.

Contribution

It introduces SRD as a novel, representation-level metric for voice anonymisation evaluation, revealing privacy leaks overlooked by traditional EER-based methods.

Findings

SRD uncovers privacy leaks missed by EER metrics.

Representation-level metrics provide more nuanced privacy assessments.

SRD demonstrates potential as a flexible evaluation tool.

Abstract

The evaluation of voice anonymisation remains challenging. Current practice relies on automatic speaker verification metrics such as the equal error rate (EER). Performance estimates dependent on the classifier and operating point provide an incomplete or even misleading characterisation of privacy risk. We investigate the use of similarity rank disclosure (SRD), an information-theoretic metric, which operates on feature representations rather than classifier decisions, providing a threshold-independent assessment of privacy and analysis of both average and worst-case disclosure. We report its application to speaker embeddings, fundamental frequency, and phone embeddings using 2024 VoicePrivacy Challenge systems. The SRD reveals privacy leaks and system-specific weaknesses missed by EER-based evaluation. Findings highlight the merit of representation-level metrics and demonstrate the potential of SRD as a flexible and interpretable tool for the evaluation of voice anonymisation.