SynthForensics: Benchmarking and Evaluating People-Centric Synthetic Video Deepfakes

TL;DR

SynthForensics is a comprehensive benchmark dataset of synthetic videos that improves evaluation of deepfake detection by focusing on realistic human depiction and human-validated comparisons.

Contribution

It introduces a new benchmark dataset with paired-source videos, human validation, and multiple compression levels, addressing limitations of existing synthetic-video evaluation methods.

Findings

Raters prefer SynthForensics in 71-77% of comparisons over existing benchmarks.

Face-based detection methods' performance drops significantly on SynthForensics.

Fine-tuning detectors reduces performance gap but at a cost to legacy benchmarks.

Abstract

Modern T2V/I2V generators synthesize people increasingly hard to distinguish from authentic footage, while current evaluation suites lag: legacy benchmarks target manipulation-based forgeries, and recent synthetic-video benchmarks prioritize scale over realistic human depiction. We introduce SynthForensics, a people-centric benchmark of $20{,}445$ videos from 8 T2V and 7 I2V open-source generators, paired-source from FF++/DFD reals, two-stage human-validated, in four compression versions with full metadata. In our paired-comparison human study, raters prefer SynthForensics in $71$--$77\%$ of head-to-head comparisons against each of nine existing synthetic-video benchmarks, while facial-quality metrics fall within the FF++/DFD baseline range. Across 15 detectors and three protocols, face-based methods drop $13$--$55$ AUC points (mean $27$) from FF++ to SynthForensics and a further $23$ under aggressive compression; fine-tuning closes the gap at a backward cost on legacy benchmarks; training from scratch shows synthetic and manipulation features largely disjoint for most detectors. We release dataset, pipeline, and code.