Beyond Seeing Is Believing: On Crowdsourced Detection of Audiovisual Deepfakes

TL;DR

This study evaluates the effectiveness of crowdsourcing in detecting audiovisual deepfakes, revealing that while crowd workers are good at identifying authentic videos, they often miss manipulations and struggle with modality attribution.

Contribution

The paper provides empirical insights into crowdsourcing's potential and limitations for audiovisual deepfake detection, highlighting the challenges in manipulation recognition and modality attribution.

Findings

Crowd workers rarely misclassify authentic videos as manipulated.

Manipulation detection accuracy improves with multiple judgments but remains limited.

Identifying manipulation type, especially joint audio-video, is significantly noisier than authenticity detection.

Abstract

Deepfakes are increasingly realistic and easy to produce, raising concerns about the reliability of human judgments in misinformation settings. We study audiovisual deepfake detection by measuring how consistently crowd workers distinguish authentic from manipulated videos and, when they flag a video as manipulated, how accurately they identify the manipulation type (audio-only, video-only, or audio-video) and how consistently they report manipulation timestamps. We run two matched crowdsourcing studies on Prolific using AV-Deepfake1M and the Trusted Media Challenge (TMC) dataset. We sample 48 videos per dataset (96 total) and collect 960 judgments (10 per video). Results show that crowd workers rarely misclassify authentic videos as manipulated, but they miss many manipulations, and agreement remains limited across videos. Aggregating multiple judgments per video stabilizes the authenticity signal, but it cannot recover manipulations that most workers consistently miss. Manipulation type identification is substantially noisier than authenticity detection even when workers detect a manipulation, with joint audio-video cases being particularly hard to recognize. Overall, these findings suggest that crowdsourcing can provide a scalable screening signal for audiovisual authenticity, while reliable modality attribution remains an open challenge.