Sharp Multiple Instance Learning for DeepFake Video Detection

TL;DR

This paper introduces a novel sharp multiple instance learning framework for detecting partially manipulated DeepFake videos, effectively handling weak labels and improving detection accuracy through spatial-temporal encoding and theoretical analysis.

Contribution

The paper proposes a new S-MIL method that directly maps instance embeddings to video predictions, relieving gradient vanishing issues, and introduces a new dataset for partial DeepFake detection.

Findings

S-MIL outperforms existing methods on FFPMS and DFDC datasets.

Spatial-temporal encoding enhances detection of manipulated faces.

State-of-the-art results on single-frame DeepFake detection.

Abstract

With the rapid development of facial manipulation techniques, face forgery has received considerable attention in multimedia and computer vision community due to security concerns. Existing methods are mostly designed for single-frame detection trained with precise image-level labels or for video-level prediction by only modeling the inter-frame inconsistency, leaving potential high risks for DeepFake attackers. In this paper, we introduce a new problem of partial face attack in DeepFake video, where only video-level labels are provided but not all the faces in the fake videos are manipulated. We address this problem by multiple instance learning framework, treating faces and input video as instances and bag respectively. A sharp MIL (S-MIL) is proposed which builds direct mapping from instance embeddings to bag prediction, rather than from instance embeddings to instance prediction and then to bag prediction in traditional MIL. Theoretical analysis proves that the gradient vanishing in traditional MIL is relieved in S-MIL. To generate instances that can accurately incorporate the partially manipulated faces, spatial-temporal encoded instance is designed to fully model the intra-frame and inter-frame inconsistency, which further helps to promote the detection performance. We also construct a new dataset FFPMS for partially attacked DeepFake video detection, which can benefit the evaluation of different methods at both frame and video levels. Experiments on FFPMS and the widely used DFDC dataset verify that S-MIL is superior to other counterparts for partially attacked DeepFake video detection. In addition, S-MIL can also be adapted to traditional DeepFake image detection tasks and achieve state-of-the-art performance on single-frame datasets.