The First VoicePrivacy Attacker Challenge Evaluation Plan

TL;DR

This paper presents the evaluation plan for the VoicePrivacy Attacker Challenge, which assesses attacker systems against voice anonymization methods using a standardized metric and dataset, fostering advancements in voice privacy security.

Contribution

It introduces a new challenge framework for evaluating attacker systems against voice anonymization, including datasets, baseline systems, and evaluation metrics, to advance voice privacy research.

Findings

Establishment of a standardized evaluation protocol.

Provision of datasets and baseline attacker system.

Anticipated identification of effective attack strategies.

Abstract

The First VoicePrivacy Attacker Challenge is a new kind of challenge organized as part of the VoicePrivacy initiative and supported by ICASSP 2025 as the SP Grand Challenge It focuses on developing attacker systems against voice anonymization, which will be evaluated against a set of anonymization systems submitted to the VoicePrivacy 2024 Challenge. Training, development, and evaluation datasets are provided along with a baseline attacker system. Participants shall develop their attacker systems in the form of automatic speaker verification systems and submit their scores on the development and evaluation data to the organizers. To do so, they can use any additional training data and models, provided that they are openly available and declared before the specified deadline. The metric for evaluation is equal error rate (EER). Results will be presented at the ICASSP 2025 special session to which 5 selected top-ranked participants will be invited to submit and present their challenge systems.