Trusted Media Challenge Dataset and User Study

TL;DR

This paper introduces the Trusted Media Challenge Dataset, a large collection of manipulated videos for fake media detection, and reports a user study showing AI models outperform humans in identifying fake media.

Contribution

The paper releases a new dataset with diverse fake media examples and provides a user study comparing human and AI performance in fake media detection.

Findings

AI models outperform humans in fake media detection

The dataset can fool human participants in many cases

The dataset supports further research in fake media detection

Abstract

The development of powerful deep learning technologies has brought about some negative effects to both society and individuals. One such issue is the emergence of fake media. To tackle the issue, we have organized the Trusted Media Challenge (TMC) to explore how Artificial Intelligence (AI) technologies could be leveraged to combat fake media. To enable further research, we are releasing the dataset that we had prepared from the TMC challenge, consisting of 4,380 fake and 2,563 real videos, with various video and/or audio manipulation methods employed to produce different types of fake media. All the videos in the TMC dataset are accompanied with audios and have a minimum resolution of 360p. The videos have various durations, background, illumination, and may contain perturbations that mimic transmission errors and compression. We have also carried out a user study to demonstrate the quality of the TMC dataset and to compare the performance of humans and AI models. The results showed that the TMC dataset can fool human participants in many cases, and the winning AI models of the Trusted Media Challenge outperformed humans. The TMC dataset is available for research purpose upon request via tmc-dataset@aisingapore.org.