Robust Localization of Partially Fake Speech: Metrics and Out-of-Domain Evaluation

TL;DR

This paper critically evaluates partial audio deepfake localization, highlighting the limitations of current metrics like EER, and proposes a more comprehensive evaluation framework emphasizing out-of-domain performance and real-world applicability.

Contribution

It reframes the localization task as sequential anomaly detection and advocates for threshold-dependent metrics, revealing the poor generalization of current models to out-of-domain data.

Findings

Current models perform well in-domain but poorly out-of-domain.

Over-optimizing for in-domain EER can harm real-world performance.

Adding partially fake data to training improves detection accuracy.

Abstract

Partial audio deepfake localization poses unique challenges and remain underexplored compared to full-utterance spoofing detection. While recent methods report strong in-domain performance, their real-world utility remains unclear. In this analysis, we critically examine the limitations of current evaluation practices, particularly the widespread use of Equal Error Rate (EER), which often obscures generalization and deployment readiness. We propose reframing the localization task as a sequential anomaly detection problem and advocate for the use of threshold-dependent metrics such as accuracy, precision, recall, and F1-score, which better reflect real-world behavior. Specifically, we analyze the performance of the open-source Coarse-to-Fine Proposal Refinement Framework (CFPRF), which achieves a 20-ms EER of 7.61% on the in-domain PartialSpoof evaluation set, but 43.25% and 27.59% on the LlamaPartialSpoof and Half-Truth out-of-domain test sets. Interestingly, our reproduced version of the same model performs worse on in-domain data (9.84%) but better on the out-of-domain sets (41.72% and 14.98%, respectively). This highlights the risks of over-optimizing for in-domain EER, which can lead to models that perform poorly in real-world scenarios. It also suggests that while deep learning models can be effective on in-domain data, they generalize poorly to out-of-domain scenarios, failing to detect novel synthetic samples and misclassifying unfamiliar bona fide audio. Finally, we observe that adding more bona fide or fully synthetic utterances to the training data often degrades performance, whereas adding partially fake utterances improves it.