TwinShift: Benchmarking Audio Deepfake Detection across Synthesizer and Speaker Shifts

TL;DR

TWINSHIFT is a new benchmark designed to evaluate the robustness of audio deepfake detectors when faced with unseen synthesis methods and speaker variations, highlighting current limitations and guiding future improvements.

Contribution

We introduce TWINSHIFT, a comprehensive benchmark for assessing audio deepfake detection robustness across unseen synthesizers and speakers, addressing a critical gap in current evaluation methods.

Findings

Current detectors struggle with unseen synthesis methods and speakers.

TWINSHIFT reveals significant robustness gaps in existing systems.

Benchmark provides guidance for developing more reliable detection methods.

Abstract

Audio deepfakes pose a growing threat, already exploited in fraud and misinformation. A key challenge is ensuring detectors remain robust to unseen synthesis methods and diverse speakers, since generation techniques evolve quickly. Despite strong benchmark results, current systems struggle to generalize to new conditions limiting real-world reliability. To address this, we introduce TWINSHIFT, a benchmark explicitly designed to evaluate detection robustness under strictly unseen conditions. Our benchmark is constructed from six different synthesis systems, each paired with disjoint sets of speakers, allowing for a rigorous assessment of how well detectors generalize when both the generative model and the speaker identity change. Through extensive experiments, we show that TWINSHIFT reveals important robustness gaps, uncover overlooked limitations, and provide principled guidance for developing ADD systems. The TWINSHIFT benchmark can be accessed at https://github.com/intheMeantime/TWINSHIFT.