Distilled Transformers with Locally Enhanced Global Representations for Face Forgery Detection

TL;DR

This paper introduces a novel hybrid transformer network with local and global feature enhancement, a mixture of experts, and self-distillation for improved face forgery detection, outperforming existing methods across multiple datasets.

Contribution

It proposes a distilled transformer architecture with locally-enhanced global representations, a mixture of experts, and a lightweight attention module, addressing issues like attention collapse and soft label scarcity.

Findings

Outperforms state-of-the-art on five deepfake datasets.

Effectively captures local and global forgery traces.

Reduces attention collapse with lightweight modules.

Abstract

Face forgery detection (FFD) is devoted to detecting the authenticity of face images. Although current CNN-based works achieve outstanding performance in FFD, they are susceptible to capturing local forgery patterns generated by various manipulation methods. Though transformer-based detectors exhibit improvements in modeling global dependencies, they are not good at exploring local forgery artifacts. Hybrid transformer-based networks are designed to capture local and global manipulated traces, but they tend to suffer from the attention collapse issue as the transformer block goes deeper. Besides, soft labels are rarely available. In this paper, we propose a distilled transformer network (DTN) to capture both rich local and global forgery traces and learn general and common representations for different forgery faces. Specifically, we design a mixture of expert (MoE) module to mine various robust forgery embeddings. Moreover, a locally-enhanced vision transformer (LEVT) module is proposed to learn locally-enhanced global representations. We design a lightweight multi-attention scaling (MAS) module to avoid attention collapse, which can be plugged and played in any transformer-based models with only a slight increase in computational costs. In addition, we propose a deepfake self-distillation (DSD) scheme to provide the model with abundant soft label information. Extensive experiments show that the proposed method surpasses the state of the arts on five deepfake datasets.