DAST: A Dual-Stream Voice Anonymization Attacker with Staged Training

TL;DR

This paper introduces DAST, a dual-stream attacker with staged training that effectively breaks voice anonymization by leveraging spectral and self-supervised features, achieving state-of-the-art attack performance.

Contribution

The paper proposes a novel dual-stream attacker with a three-stage training strategy to improve privacy evaluation of voice anonymization systems.

Findings

Stage II training enhances cross-system robustness.

Fine-tuning with 10% data surpasses current state-of-the-art.

The attacker effectively exposes vulnerabilities in anonymization methods.

Abstract

Voice anonymization masks vocal traits while preserving linguistic content, which may still leak speaker-specific patterns. To assess and strengthen privacy evaluation, we propose a dual-stream attacker that fuses spectral and self-supervised learning features via parallel encoders with a three-stage training strategy. Stage I establishes foundational speaker-discriminative representations. Stage II leverages the shared identity-transformation characteristics of voice conversion and anonymization, exposing the model to diverse converted speech to build cross-system robustness. Stage III provides lightweight adaptation to target anonymized data. Results on the VoicePrivacy Attacker Challenge (VPAC) dataset demonstrate that Stage II is the primary driver of generalization, enabling strong attacking performance on unseen anonymization datasets. With Stage III, fine-tuning on only 10\% of the target anonymization dataset surpasses current state-of-the-art attackers in terms of EER.