Performance Decay in Deepfake Detection: The Limitations of Training on Outdated Data

TL;DR

Deepfake detection models perform exceptionally well on current data but rapidly decline in accuracy as deepfake technology advances, highlighting the need for ongoing data updates and improved frame-level analysis.

Contribution

This paper demonstrates the rapid performance decay of deepfake detectors over time and emphasizes the importance of continuous data curation and frame-level features for robustness.

Findings

Models trained on recent data achieve over 99.8% AUROC.

Performance drops over 30% when tested on deepfakes from six months later.

Static, frame-level artifacts are key to detection, not temporal inconsistencies.

Abstract

The continually advancing quality of deepfake technology exacerbates the threats of disinformation, fraud, and harassment by making maliciously-generated synthetic content increasingly difficult to distinguish from reality. We introduce a simple yet effective two-stage detection method that achieves an AUROC of over 99.8% on contemporary deepfakes. However, this high performance is short-lived. We show that models trained on this data suffer a recall drop of over 30% when evaluated on deepfakes created with generation techniques from just six months later, demonstrating significant decay as threats evolve. Our analysis reveals two key insights for robust detection. Firstly, continued performance requires the ongoing curation of large, diverse datasets. Second, predictive power comes primarily from static, frame-level artifacts, not temporal inconsistencies. The future of effective deepfake detection therefore depends on rapid data collection and the development of advanced frame-level feature detectors.