TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection

TL;DR

TalkingHeadBench is a comprehensive benchmark and dataset for evaluating the robustness and generalization of deepfake talking-head detection methods against the latest generative models, addressing current limitations in existing benchmarks.

Contribution

It introduces a new multi-modal benchmark with diverse datasets and protocols to assess detection models' robustness and generalization to advanced deepfake generators.

Findings

Existing detectors show limited robustness to new generators.

Transformers outperform CNNs in detection accuracy.

Error analysis reveals common failure modes and biases.

Abstract

The rapid advancement of talking-head deepfake generation fueled by advanced generative models has elevated the realism of synthetic videos to a level that poses substantial risks in domains such as media, politics, and finance. However, current benchmarks for deepfake talking-head detection fail to reflect this progress, relying on outdated generators and offering limited insight into model robustness and generalization. We introduce TalkingHeadBench, a comprehensive multi-model multi-generator benchmark and curated dataset designed to evaluate the performance of state-of-the-art detectors on the most advanced generators. Our dataset includes deepfakes synthesized by leading academic and commercial models and features carefully constructed protocols to assess generalization under distribution shifts in identity and generator characteristics. We benchmark a diverse set of existing detection methods, including CNNs, vision transformers, and temporal models, and analyze their robustness and generalization capabilities. In addition, we provide error analysis using Grad-CAM visualizations to expose common failure modes and detector biases. TalkingHeadBench is hosted on https://huggingface.co/datasets/luchaoqi/TalkingHeadBench with open access to all data splits and protocols. Our benchmark aims to accelerate research towards more robust and generalizable detection models in the face of rapidly evolving generative techniques.