Texture, Shape, Order, and Relation Matter: A New Transformer Design for Sequential DeepFake Detection

TL;DR

This paper introduces TSOM, a novel Transformer architecture for sequential DeepFake detection that leverages texture, shape, and order cues, with an extended version TSOM++ that also models manipulation relations, achieving superior performance.

Contribution

The paper proposes a new Transformer design, TSOM, with four key innovations for improved sequential DeepFake detection, and extends it to TSOM++ by incorporating manipulation relation modeling.

Findings

TSOM outperforms existing methods on benchmark datasets.

The texture-aware branch effectively captures subtle manipulation traces.

Modeling manipulation relations with contrastive learning further improves detection.

Abstract

Sequential DeepFake detection is an emerging task that predicts the manipulation sequence in order. Existing methods typically formulate it as an image-to-sequence problem, employing conventional Transformer architectures. However, these methods lack dedicated design and consequently result in limited performance. As such, this paper describes a new Transformer design, called {TSOM}, by exploring three perspectives: Texture, Shape, and Order of Manipulations. Our method features four major improvements: \ding{182} we describe a new texture-aware branch that effectively captures subtle manipulation traces with a Diversiform Pixel Difference Attention module. \ding{183} Then we introduce a Multi-source Cross-attention module to seek deep correlations among spatial and sequential features, enabling effective modeling of complex manipulation traces. \ding{184} To further enhance the cross-attention, we describe a Shape-guided Gaussian mapping strategy, providing initial priors of the manipulation shape. \ding{185} Finally, observing that the subsequent manipulation in a sequence may influence traces left in the preceding one, we intriguingly invert the prediction order from forward to backward, leading to notable gains as expected. Building upon TSOM, we introduce an extended method, {TSOM++}, which additionally explores Relation of manipulations: \ding{186} we propose a new sequential contrastive learning scheme to capture relationships between various manipulation types in sequence, further enhancing the detection of manipulation traces. We conduct extensive experiments in comparison with several state-of-the-art methods, demonstrating the superiority of our method. The code has been released at https://github.com/OUC-VAS/TSOM.